Hi!

I'm Parth, and this is the public component of my second brain.

I'm a programmer with many interests, but I'm primarily working on my startup Lockbook.

In the General Programming section you can find some abstract ideas about building products, my latest position various programming topics, and other areas of exploration.

Lockbook is secure note taking platform. If you'd like to learn more about it checkout our website. There you can learn more about the product, see some live demos, and dive deeper into Lockbook's product documentation. The Lockbook section here captures some of the aspects of the project I'm most proud to have contributed.

I like working with my hands, the wrenching section catalogues my journey with various motorcycles, my project miata, and various other physical world pursuits.

Finally Life Tracking is a place I can share things like what books I'm reading, how I organize my spaces, and anything else that's captured my interest.

Some handy links:

• Github: @Parth
• Instagram: @MehrotraParth
• Youtube: @ParthMehrotra
• Spotify

This whole site is literally just a folder in my Lockbook that I edit as I experience existence. A tiny Github Actions script builds an MdBook and publishes it to Github Pages.

How do you see the world

I've been giving a lot of thought to the various ways people view problems and solutions, particularly: how does what you spend most of your time doing map to the way you view the world? For example, people who spend a lot of time thinking about marketing and sales may have the following perspective: all the problems and solutions are out there; the hard part is creating a narrative that connects the dots. On the other hand, I would wager that people in finance see things in terms of accurate pricing, forecasting, and resource allocation.

Of course, the thing that influences your perspective doesn't have to be tied to your profession (or any profession for that matter), but it likely does need to be tied to whatever you're putting your 10,000 hours into. I think this starts to happen somewhere around the time you consider the thing to be a central part of your identity. It's likely tied to the point where you start to form your own opinions using expertise and specific knowledge that other people don't have. At this point, you're in a position to contribute something novel back, hopefully something of value.

Personally, I'm very interested in putting my 10,000 hours towards advancing the state of technology. I find advancing technology (particularly writing software) to be intrinsically rewarding, I'd happily do this in a vacuum. However we're not a vacuum, and while every problem is not technical in nature, I see many that are.

Now, I haven't crossed hour 10,000 yet, but I'm ready to start sharing my ideas and the way I think about certain things. I've learned a lot so far and I've formed a more clear idea of the type of technology I'm interested in advancing: technology that empowers the individual.

I want to use this blog to detail the things I'm exploring, share what I value and the potential I see in the future.

Ideation

Over the last 10 years I've gone through the process of ideation, execution and delivery countless times. I'd like to share some lessons I've learned and how I think about projects today.

The Idea

It's hard to evaluate how important an idea is relative to execution and presentation.

I think ideas are a dime a dozen, and most projects fail not because the idea was bad, but because an attempt was never made.

However, you should expect good ideas to be rare. I like using challenge oriented thinking to filter out bad ideas:

First filter: does this exist already? It may exist, and be popular, and maybe you just give that product or service a try. Maybe it doesn't exist because it's a foolish idea that would never work. Or maybe it doesn't exist because no one's made it yet. If you don't make stuff, there is no stuff. Perhaps bringing this thing into existence is your calling.

Maybe your idea is out there already, you give it a try and have a bad experience. When you're ready to give up on this product, you wonder to yourself: "Can I do a better job?". You have to evaluate how much your execution edge is. Do you have the expertise and resources to do this better? Have you overlooked something that makes this very hard? Peter Thiel's Zero to One explores this idea in a bit more depth. In short he provides the following intuition: unless your execution is orders of magnitude better than your competition you're better of seeking novel ideas that have a higher likelihood of earning you a monopoly.

Once this "market research" filter has been overcome, I look to gain perspective from people I respect. Traits I look for in people to discuss ideas with: generally capable people, well-read, and open minded. I want objective feedback, but I'm not too interested in knee-jerk reactions. My favorite conversations are ones where we try to build up the idea then see if we can tear it apart.

After this, I just let the idea sit in a document next to other ideas. Unless there's a unique opportunity, I'm primarily interested in working on things that would help me in some way. Once it lives in my notebook alongside other ideas, whenever I next feel the need for this I'll make a note. As I'm going through life, I'll put all associated thoughts into this document. My time is limited and this gives me an organic way for my idea to mature and allows me to filter out ideas that seemed interesting in the moment but wouldn't really provide me or anyone any real value.

By the time I sit down to start executing on an idea, I'm usually fed up with whatever the stop gap I've been using is, and I'm in a mental state where I don't really care if other people will use this thing I want it to exist for me. In my experience this makes the process of willing the idea into existence a transcendent experience. You'll understand your needs best and stop over-prioritizing the opinion of other people. The basis for a good MVP is the product you would use, not the hackathon power point presentation you're trying to pass off as your product.

Once you have a core product you're proud of then you can consider if you want to solicit Lean Startup style feedback from the market and your customers or if you want to continue to take a Job's style visionary approach to your product.

System Design

When I first started exploring programming I wasn't sure how anything was made. As I learned about the various components of a stack, I started to form a clearer picture of the ways people design systems.

For instance, when I learned about if statements and strings, I understood how people validated forms. And when I learned how to process stdin and output to stdout I understood how something like grep is made. I didn't necessarily understand how to write a regex parser, but I had a good sense of how CLIs come together.

As my understanding matured, I had a sense about why certain designs were successful as well as what we may expect from system design in the future. I'd like to share a snapshot of my current understanding.

When I'm trying to conceptualize the solution to a problem, I start by trying to figure out what shape my work will take. Usually, this is one of the following:

1. Self-standing executable
2. Classic web app
3. Complex web app
4. Novel architecture

Self Standing executable

A self-standing executable is a piece of software that doesn't have critical external dependencies.

This generally takes the shape of:

• a static site (like this one)
• a mobile app
• a CLI
• a library
• a bot
• some games

These projects are the building blocks of our computing infrastructure. They take a look at a very specific problem and attempt to solve it excellently. There are some unique aspects of such projects:

Software in this category has particularly low maintenance costs. If the solution requires you to maintain a server, it vastly diminishes the long-term survivability of a project. For instance, CLI's (like grep or vim) or libraries (like openssl or TensorFlow) don't require the upkeep of centralized services. While they may evolve and improve, you can probably use a variant from 10 years ago without any issue. And you can probably use that same variant 10 years from now. I would be surprised if the Skype client from 5 years ago worked properly today. It certainly wouldn't work if Skype's servers went offline. On the other hand, how would you go about "killing" vim, openssl, and similar projects?

Because you don't have to worry about things like maintaining servers, you also don't need to worry about scaling. Static sites are distributed via CDNs, CLIs via package managers, and apps in the App Store. Depending on the idea, you expose yourself to the possibility that your application is virally adopted.

Finally, your ability to prove that your implementation is correct dramatically decreases with the amount of code involved. I'm far more confident in my ability to play audio files locally, than my ability to stream music to my Bluetooth speaker. Consider the difference in the amount of code at play for those examples. Doing things locally, simply, and efficiently reduces the surface area of possible problems.

I think there's a temptation here to monetize this sort of application. This could look something like making people sign up for accounts when that provides no benefit to the experience, or serving them ads. What you sacrifice for doing such things is the possibility for your creation to be widely adopted by all sorts of people across a large period. I doubt git would have reached where it is today if you needed to create an account on Linus Torvald's website. Linus Torvalds is living a very comfortable life. I think it's in your best interest to remove as many barriers to mass adoption and create the best product you can. If you do a good job, you'll be okay.

Classic web app

For a solution to fall into this category it usually necessitates that users create accounts with a centralized service to solve their problem. For instance, social networks, messengers, and banks fundamentally depend on their ability to authenticate users.

These sort of services generally involve:

• Frontend clients: websites, mobile apps, CLI's
• API Servers: stateless, horizontally scalable
• Store of state: some DB which will depend on the type of data and how you access it

There are several technical challenges the come from these additional moving parts. But at this stage, the patterns are still pretty well explored. At some point, your website may become slow because you're getting too much traffic. You'll have to transition from 1 API server to many API servers, you'll likely have to use some sort of a load balancer. It'll be tricky for sure, but lots of people have explored this before you and there shouldn't be too many unknowns after you consume the right resources and ask the right questions.

The most common mistake in this realm is creating un-needed complexity. This could be a premature optimization (involving caches like Redis too early), or could involve splitting their API layer into "microservices". I'll explore this particular mistake in a future post, but the main idea I'd like to stress is: be skeptical of additions to your architecture that stray too far from this model.

Complex web app

Scaling a classic web app is straightforward because it's known and generally involves adding/upgrading hardware. But sometimes this isn't enough, and your performance needs require you to rethink your architecture. Generally, this happens because the time it takes for various computers to talk to each other introduces too much latency, or because you can't mutate your state (if it's stored in a DB) quickly enough. Generally what you'll end up doing is taking the component that's proving difficult to scale and try to solve the problem with a single computer. Keeping state in RAM instead of on disk, and communicating via function calls rather than network requests.

Let's consider an example: popular games like Dota, Fortnight, Overwatch have to worry about matchmaking, running games, managing item ownership, and processing payments. Everything apart from administering the game logic can likely be engineered as a classic web app. But managing the state of the game (where players are, who did what, who's winning) has tight real-time requirements. By creating a "stateful node" you may be able to meet these real-time requirements, but you have likely created a new set of problems.

Even though you've solved the performance issues that were caused by a database, it's still not clear how you would scale this stateful node to thousands of users. You'll likely need some sort of flexible infrastructure that will let you spin up these game servers on demand.

It's also not obvious how you would achieve high availability. What happens when you want to update your stateful node? In the example of the game, this isn't a significant problem: games are short-lived, new games can start on new versions of the code. However, how do you go about updating the matching engine of an exchange? You can take downtime and lose money, or you invest significant engineering resources to design a matching engine that operates and shares state with multiple nodes and can be upgraded incrementally.

Making this component fault-tolerant can be a non-trivial task as well. In a classic web app, if one node fails you can fall back to the other ones, there was no state in that node, so nothing of importance was lost. However, if your game server experiences a failure, those players likely just disconnected (and are very upset).

This existence of a "stateful node" is just one type of architectural element that turns a "Classic" web app into a "Complex" one. Others can include cron jobs, data pipelines, or specialized hardware. Creating a specialized service that performs well, is highly available and fault-tolerant will likely require a lot of domain-specific knowledge. You won't find a community or textbooks full of answers. You'll likely just have to try several things and learn from experience. Every success, however, becomes a technical edge.

Novel architecture

Sometimes your problem domain imposes a strong constraint that requires something that hasn't been seen before.

Decentralized projects like Bitcoin, Bittorrent, and Tor are worth studying. All these projects are relatively old, Bitcoin is 10+ years old at the time of writing. The solution that Bitcoin promises: store of value, payment system, created outside the cathedral of big banks and big government had been tried many times before. But due to the way previous attempts were designed, they were shut down. Similarly, early file-sharing apps that were used to pirate music were trivially shut down as well. The problems these tools are trying to solve remained largely unsolved until their decentralized variants emerged.

Evolution of design

Today's systems look this way because of our prior expectations from our computing infrastructure, as well as the software and hardware limitations of the previous generation of softare.

For instance, it's hard to create a P2P application because of the NAT tables IPv4 necessitates. Addressing devices that don't have fixed IP addresses (all consumer devices) is difficult and requires workarounds like STUN and TURN. New technologies on the horizon as well as an increased appetite for decentralized technologies could lead to a proliferation of P2P applications.

To me, the intersection of our shifting desires (performance, censorship resistance, privacy, etc) and new technologies on the horizon is one of the most exciting areas of our craft and that is why I think these are some of the most valuable problems to work on.

Microservices FAQ

The world is too eager to use microservices. Motivations for splitting a program out into independent servers are on shaky ground. This results in needlessly lower productivity and lower quality software.

What is a microservice architecture?

An architecture in which several networked small (micro) programs (services) communicate to perform a task. For example, a streaming service may have independent services for Accounts, Recommendations, Billing and Streaming.

What is a monolith?

A single program which contains all the logic to perform a task. For example, a streaming service may have a monolith with a route for /accounts, /recomendations, /billing and /streaming.

What is strong evidence something should be a microservice?

When the hardware requirements or performance characteristics of your program deviate significantly from your monolith. If your monolith is a traditional, statless, horizontally scalable, REST API and you require the ability to spin up n number of stateful game servers for each group of 10 people playing your game, you should probably spin those games up as independent services.

What is the productivity cost of an unnecessary microservice?

Reasoning about a distributed system is hard. If you needlessly split up your operations into independent services each with their own database you're handicapping your ability to make illegal states unlikely or impossible.

You're also usually taking calls that would be simple function calls checked by the compiler and turning them into network requests. In the best case this forces you to use tools like grpc, protobuffers. In the worst case this reduces the reliability of your operations. In either case you're going to spend much more time reasoning about how to roll out breaking changes to a distributed system that would otherwise be light refactors.

Infrastructure related productivity abstractions are hard. Your org will likely want a suite of support tools for each service, these can include: monitoring, load-balancing, logging, alerting, etc. You're forcing your company into one of 3 outcomes:

1. hire more operations staff
2. lock yourself into a cloud ecosystem
3. adopt tool(s) like Terraform, Ansible, and other SRE orchestration software.

Is a monolith harder to scale?

No, you can run n replicas of your monolith to achieve the same scale characteristics as your microservices.

What if I want to independently scale different portions of my app?

You can configure a reverse proxy to only send traffic from a particular route to a particular node, achieving the same flexibility.

Isn't it about a separation of concern?

You should organize code to separate concerns. Your API framework probably supports the idea of routes, your language probably supports modules, packages or libraries.

Isn't a microservice architecture more resiliant to failures?

Most architectures have several single points that if failed would result in a total system outage. Generally this is your database. Reasoning about multiple databases is usually not worthwhile and causes more outages than it prevents. More often than not the components of a microservice depend on each other, and there are very few "unimportant" components. In the streaming example, it's very likely that lots of interactions will cause you to engage with the Account service to see if a user exists, or the Billing service to see if they're allowed to watch something. If there's a significant defect in any major component, it's usually a total outage for most customers.

Microservices don't reduce the need for strong engineering processes that would catch defects (like docs, tests, pr reviews, and other forms of QA).

Microservices allows team A to work indepenedently of team B

This is usually an organizational problem microservices is hiding.

If team A and team B are working on independent components then each team should be able to make changes to their portion of the monolith and deploy whenever they'd like.

If the two teams are consuming each other's services, then they still need to give the same level of attention to breaking changes. And similar analogues apply whether that interaction is happening over the network or via a function call.

Microservices allow my team to deploy more often

The only reason your company shouldn't deploy code that's gone through all the QA processes is if the service is stateful and no level of downtime is permissible due to hard engineering constraints. If you're unable to deploy a stateless rest api without downtime, that should almost certainly become a high engineering priority, and this isn't a good reason to build a microservice in the meantime.

There are services that are too risky to update

Services that are too risky to update are probably too risky to exist. Microservices here are a bandaid on a problem to which the solution is better engineering processes.

Microservices allow my company to use different languages

Using different languages has a high cognitive cost, and has some huge missed opportunities for engineering culture.

Different teams have different priorities, the wallet team at a crypto exchange may care about security, while the exchange team may care about performance.

Having these ideas expressed in a single language allows junior engineers to understand contrasting values more clearly. It promotes cross-team collaboration and stewardship rather than silos of ownership.

Using a single language allows teams to share code, learnings, and expertise across the company.

There should be incredibly tangible reasons for deviating from an organizations primary language, like the availability of specific tooling that delivers a massive ROI, or the infeasibility of sharing a language across two dramatically different modalities of thinking (frontend / backend and the cost of novel solutions is unacceptable).

More often than not, however multiple languages without strong reasons indicates to me a failure of engineering leadership to define clear values.

A new note-taking app

Many moons ago my friends and I found ourselves frustrated with our computing experience. Our most important tools: our messengers, our note-taking apps, and our file storage all seemed to leave us with the same bitter taste in our mouths. Most of the mainstream solutions were trapped inside a browser, the epitome of sacrificing quality for the lowest common denominator. They didn't pass muster for basic security and privacy concerns. Building on top of these systems was a painful experience. We were also growing concerned that we did not share the same ethics and values as the large companies running these platforms.

So we ventured out into the land of FOSS. We built our systems from the ground up and experimented with self-hosted solutions. While we learned a lot from this experience, these solutions didn't stand the test of time. As we brought our friends into this world, we found ourselves constantly apologizing for the sub-par experience. It felt like we solved some of our problems, but made other ones (UX) worse.

At this point, we took a step back. I knew we could do better. The apps we were being critical of weren't some of the fresher ideas in computing. They are things that we've been doing for decades. Why don't these products feel more mature? What should software that's been around this long feel like?

What is ideal?

Software that's been around this long shouldn't be trapped in the browser. A browser is a convenient place for the discovery of new information, it's not the place I want to visit for heavily used, critical, applications. When I look at my devices, whether on my iPhone or my Linux laptop, the apps I can use with the least friction are simple, native applications. They have the largest context about the device I'm using encoded into the application. This friction-free experience is why people reach for Apple Notes on their iPhones. And when they open those same notes on their iPad they find rich support for their Apple Pencil. For me, a minimal, friction-free context-aware experience is more valuable than feature richness.

Whatever experience I have on one device should carry over seamlessly to any device I may end up owning in the future. My notes shouldn't be trapped on Apple Devices should I want to transition to Linux. Very few actions should require a network connection, and any network interactions should be deferrable so people outside of metropolitan areas don't have a poor experience.

For now, likely most of these services will have to interact with some sort of backend. Everything that backend receives should be encrypted by the customer themselves, in a manner that nobody besides them and the people they give access to can see that content. We shouldn't ask the customer for any information the service doesn't require. There is very little reason that a user must provide an email address or a phone number to use a note-taking app. This level of security and privacy shouldn't cost the user anything in terms of quality. Our customers may be whistle-blowers, journalists, or citizens living under oppressive regimes. They simply cannot afford to trust and they shouldn't have to.

This software is too important to not open source. Any software claiming to be secure needs to be open source to prove that claim. Sensitive customers need the ability to build minimal clients with small dependency trees from sources on secure systems. Open-sourcing components like your server signal to the world that they can host critical infrastructure themselves, even if the people behind the product lose the will to keep the lights on. Open source doesn't end in making the source code available, this software should be built out in the open with help from an enthusiastic community. People should be able to extend the tools for fun or profit with minimal friction.

Reaching for an ideal

After much discussion, we decided that the best place to start was a note-taking app. We felt it was the product category with the largest room for growth. Architecturally it also paves the way for us to tackle storing files. And so began the three-year-long journey to create Lockbook a body of work I'm proud to say has stayed true to the vision outlined above. At the moment, Lockbook is not quite ready for adoption as it's in the early stages of alpha testing. But I'd like to use this space to share updates on our progress as well as document how we overcame some interesting engineering challenges like:

• Productively maintaining several native apps with a small team
• How we create rich non-web cross-platform UI elements.
• How we leverage a powerful computer to find bugs.

If you'd like to learn more about Lockbook you can:

• Checkout our website
• Browse our source code
• Join our discord

If you'd like to take an early look at Lockbook, we're available on all platforms.

• Github Releases
• Apple App Store
• Google Play
• Brew
• AUR
• Snap

Why Lockbook chose Rust

Lockbook began it’s journey as a bash script. As it started to evolve into something more serious, one of our earliest challenges was identifying a UI framework we were willing to bet on. As we explored, we were weighing things like UI quality, developer experience, language selections, and so on.

Our choice of UI framework had implications for our server as well. If we chose JavaFX and native Android, we would likely want to choose a JVM-based language for our server to share as much code as possible. 

As we wrote and re-wrote our application, we discovered that most of our effort, even on our clients, was not front-end code. When we were implementing user management, billing, file operations, collaboration, compression, and encryption, the lion’s share of the work was around traditional backend-oriented tasks. Things like data modeling, error propagation, managing complex logic, handling database interactions, and writing tests were where we were spending most of our time. Many of these things had to take place on our clients because all our user data is end-to-end-encrypted. Additionally, some of these operations were sensitive to slight differences in implementation. If your encryption and decryption are subtly different across two different clients, your file may be unreadable.

It was also becoming clear to us that the applications that looked and felt the best to us were created in that platform’s native UI framework. So our initial investigation around UI frameworks morphed into an inquiry into what the best repository for business logic was. Ideally, this repository would give us great tools for writing complex business logic and would be ultimately portable.

Tools for managing complexity

Our collective experience made us gravitate towards a particular spirit. At Gemini, Raayan and I saw how productive we were within a foreign, large-scale, Scala codebase. Informed by the experience we were looking for a language with an expressive, robust type system.

A “robust type system” goes beyond what you’d find in languages like Java, Python, or Go. We were looking for type systems where null or nil were the exception, rather than the norm. We want it to be apparent when a function could return an error, or an empty value, and have ergonomic ways to handle those scenarios.

We wanted to have sophisticated interactions with things like enums, specifically, we wanted to be able to model the idea of exhaustivity. When an idea we were working with evolved to have more cases we wanted our compiler to guide us to all the locations that need to be updated.

There were a handful of other features we were looking for which can broadly be categorized into two similar ideas: 

We wanted to express as much as we could in our primary programming language. Things that would traditionally be documented (this fn will return null in this situation) or things that would be expressed in configuration (TLS configuration handled by a different program in YAML) would ideally be expressed in a language that contributors understood intimately. Ideally in a language where the compiler was providing strong guarantees against mistakes and misuse.

We wanted our language and tools to help us detect defects as early as possible in the development lifecycle. Most software developers are used to trying to capture defects at test time, but we found that trying to capture defects even earlier, at compile time, allowed us to drop into flow more easily. The following is our preference for when we’d like to catch defects:

1. at compile time
2. test time
3. startup time
4. pr time
5. internal test time
6. by a customer

Our strongest contenders for languages here were Rust, Haskell, and Scala.

Ultra-portability

Ideally, this repository would not place constraints on where it could be used. If our repository was in Scala, for instance, we’d be able to use it on Desktop, our Server, and Android, but we’d run into problems on Apple devices.

We could use something like JS, virtually every platform has a way to invoke some sort of WebView which allows you to execute JS. But we’d had plenty of bad experiences with vanilla javascript. We found that evolutions on JS like Typescript were also on a shaky foundation. Despite the JS ecosystem being popular and old, it didn’t feel very mature. Finally, we didn’t like the way most js-based applications, whether Electron or React Native felt.

Both JS and Scala would require tremendous overhead due to the default environments in which they run. We needed something lighter weight than invoking a little browser every time we wanted to call into our core. Our team members were pretty experienced in Golang, and Cgo was an ideal fit for what we were looking for. It would allow us to ship our core as a C library accessible from any programing language we were interested in inter-operating with. There were some concerns we had about the long-term overhead of cgo and garbage collection generally, but those wouldn’t be immediate concerns.

Similarly, Rust had a pretty rich collection of tools for generating C bindings for Rust programs and a pretty mature conceptualization of FFI. Though it wasn’t an immediate criterion we were inspired by the fact that most everything in Rust was a zero-cost abstraction. In that spirit, FFI in Rust would have virtually no additional overhead when compared to a C program. We were also drawn to Cargo which felt like the package manager for a language we were waiting for, particularly useful for our complicated build process.

Our strongest contenders for languages here were Rust, Go, and C.

Taking the plunge

Learning Rust wasn’t a smooth process, but solid documentation helped us overcome the steep learning curve. Every language I’ve learned so far has shaped the way I view programming, it was refreshing to see the interaction of high-level concepts like Iterators, Dynamic Dispatch, and pattern matching discussed alongside their performance implications.

Rust has an interesting approach to memory management: it heavily restricts what you can do with references. In return, it will guarantee all your references are always valid and free of race conditions. It will do this at compile time, without the need for any costly runtime abstraction like Garbage Collection.

Once we were over the learning curve we prototyped the core library we’d been planning, a CLI, and a Server that used it. During a period when many of us were rapidly prototyping many different solutions, this was the one that stood the test of time. Soon after the CLI, a C binding followed, then an iOS and macOS application. Today we have a JNI bindings and an Android app as well. This core library will one day be packaged and documented as the Lockbook SDK allowing you to extend Lockbook from any language (more on this later).

Further personal reflections

You can probably predict what your experience with Rust is going to be based on how you felt about the above two priorities. Rust is an experiment in the highest-level features implemented at no runtime cost. If you feel like the Option<T> is not a useful construct, you’re not likely to appreciate waiting for the compiler. If you don’t mind the latency introduced by garbage collection you’re not going to enjoy wrestling the borrow checker. 

I wasn’t specifically seeking out performance, but before Rust, while programming there was always a slight uncertainty about whether I would have to re-write a given component in C, or spend time tuning a garbage collector. In Rust I don’t write everything optimally initially, when I need to, I’ll clone() things or stick them in an Arc<Mutex<T>> to revisit at a later time, but I appreciate that all these artifacts of the productivity vs. performance trade-offs are explicitly present for me to review, rather than implicitly constrained by my development environment.

For our team, learning Rust has certainly been a dynamic in onboarding new contributors. Certainly, we’ve lost contributors who didn’t buy into the ideas and were turned away because of Rust. But we’ve also encountered people who are specifically seeking out Rust projects because they share our excitement. It’s hard to tell what the net impact here is, but as is the case every year: Rust is a a language a lot of people love. Significant Open Source and Commercial entities from Linux to AWS are making permanent investments in Rust.

This excitement does however bring a lot of Junior talent to the ecosystem, subsequently, even though it’s roughly as old as Go, many of Rust’s packages feel like they’re not ready for production. By my estimation, this is because in addition to understanding the subject matter of the package they’re creating a maintainer of a library needs to understand Rust pretty deeply. Additionally, within the Rust ecosystem, some people are optimizing for different things. Some people are optimizing for compile times and binary size, while others are optimizing for static inference and performance, in many cases these are mutually exclusive values. 

In some cases, this is a short-term problem features are stabilized, and best practices are identified. In other cases, this is an irreconcilable aspect of the ecosystem that will simply result in lots of packages that are solving the same problem in slightly different ways.

This is something we should expect, as Rust is a language that’s trying to serve all programmers from UI developers to OS designers. And though it may cost me some productivity in the short term while I’m forced to contend with this nuance, in the long term it massively broadens my horizons as a software engineer.

Personally what got me over the steep learning curve is a rare feeling that the knowledge I’m building while learning Rust is a permanent investment in my future, not a trivial detail about a flaw of the tool I’m using. I’m very excited to see where Rust takes us all in the future.

db-rs

The story of how Lockbook created its own database for speed and productivity.

As a backend engineer, the architecture I see used most commonly is a loadbalancer distributing requests to several horizontally scaled API servers. Those API servers are generally talking to one or more stores of state. Lockbook also started this way, we load balanced requests using HAProxy, had a handful of Rust API nodes, and stored our data in Postgres and S3.

A year into the project, we had developed enough of the product that we understood our needs more clearly, but we were still early enough into our journey where we could make breaking changes and run experiments. I had some reservations about this default architecture, and before the team stabilized our API, I wanted to see if we could do better.

My first complaint was about our interactions with SQL. It was annoying to shuffle data back and forth from the fields of our structs into columns of our tables. Over time our SQL queries grew more complicated, and it was hard to express and maintain ideas like a user's file tree cannot have cycles or a file cannot have the same name as a non-deleted sibling. We were constantly trying to determine whether we should express something in SQL, or read a user's data into our API server, perform and validate the operation in Rust, and then save the new state of their file tree. Concerns around transaction isolation, consistency, and performance were always hard to reason about. We were growing frusterated because we knew how we want this data to be stored and processed and were burning cycles fighting our declarative environment.

My second complaint was about how much infrastructure we had to manage. While on the topic of Postgres itself, running Postgres at a production scale is not trivial. There's a great deal of trivia you have to understand to make Postgres work properly with your API servers and your hardware. First we had to understand what features of Postgres our database libraries supported. In our case, that meant evaluating whether we needed to additionally run PGBouncer, Postgres' connection pooling server, and potentially another piece of infrastructure to manage. Regardless of PGBouncer, configuring Postgres itself requires an understanding of how Postgres interacts with your hardware. From Postgres' configuration guide:

PostgreSQL ships with a basic configuration tuned for wide compatibility rather than performance. Odds are good the default parameters are very undersized for your system....

That's just Postgres. Similar complexities existed for S3, HAProxy, and the networking and monitoring considerations of all the nodes mentioned thus far. This was quickly becoming overwhelming, and we hadn't broken ground on user collaboration, one of our most ambitious features. For a team sufficiently large this may be no big deal. Just hire some ops people to stand up the servers so the software engineers can engineer the software. For our resource-constrained team of 5, this wasn't going to work. Additionally, when we surveyed the useful work our servers were performing, we knew this level of complexity was unnecessary.

For example, when a user signs up for Lockbook or makes an edit to a file, the actual useful work that our server did to record that information should have taken no more than 2ms. But from our load balancer's reporting, those requests were taking 50-200ms. We were using all these heavy-weight tools to be able to field lots of concurrent requests without paying any attention to how long those requests were taking. Would we need all this if the requests were fast?

We ran some experiments with Redis and stored files in EBS instead of S3, and the initial results were promising. We expressed all our logic in Rust and vastly increased the amount of code we were able to share with our clients (core). We dramatically reduced our latency, and our app felt noticeably faster. However, most of that request time was spent waiting for Redis to respond over the network (even if we hosted our application and database on the same server). And we were still spending time ferrying information in and out of Redis. I knew something was interesting to explore here.

So after a week of prototyping, I created db-rs. The idea was to make a stupid-simple database that could be embedded as a Rust library directly into our application. No network hops, no context switches, and huge performance gains. Have it be easy for someone to specify a schema in Rust, and allow them to pick what the performance characteristics of these simple key-value style tables would be. This is Core's schema, for instance:

#![allow(unused)]
fn main() {
#[derive(Schema, Debug)]
pub struct CoreV3 {
    pub account: Single<Account>,
    pub last_synced: Single<i64>,
    pub root: Single<Uuid>,
    pub local_metadata: LookupTable<Uuid, SignedFile>,
    pub base_metadata: LookupTable<Uuid, SignedFile>,
    pub pub_key_lookup: LookupTable<Owner, String>,
    pub doc_events: List<DocEvent>,
}
}

The types Single, LookupTable, and List are db-rs table types. They are backed by Rust Option, HashMap, or Vec respectively. They capture changes to their data structures, Serialize those changes and append them to the end of a log -- one of the fastest ways to persist an event.

The types Account, SignedFile, Uuid, etc are types Lookbook is using. They all implement the ubiquitous Serialize Deserialize traits, so we never again need to think about converting between our types and their on-disk format. Internally db-rs uses bincode format, an incredibly performant and compact representation of your data.

What's cool here is that when you query out of a table, you're handed pointers to your data. The database isn't fetching bytes, serializing them, or sending them over the wire for your program to then shuffle into its fields. A read from one of these tables is a direct memory access, and because of Rust's memory guarantees, you can be sure that reference will be valid for the duration of your access to it.

What's exciting from an ergonomics standpoint is that your schema is statically known by your editor. It's not defined and running on a server somewhere else. So if you type db. you get a list of your tables. If you select one, then that table-type's contract is shown to you, with your keys and values. Additionally for us, now our backend stack doesn't require any container orchestration whatsoever: you just need cargo to run our server. This has been massive boon for quickly setting up environments whether locally or in production.

The core ideas of the database are less than 800 lines of code and are fairly easy to reason about. This is a database that's working well for us not because of what it does, but because of all the things it doesn't do. And what we've gained from db-rs is a tremendous amount of performance and productivity.

Ultimately this is a different way to think about scaling a backend. When you string together 2-4 pieces of infrastructure over the network, you're incurring a big latency cost, and hopefully what you're gaining as a result is availability. But are you? If you're using something like Postgres, you're also in a situation where your database is your single point of failure. You've just surrounded that database with a lot of ceremonies, and I'm skeptical that the ceremony helps Postgres respond to queries faster or that it helps engineers deliver value more quickly.

db-rs has been running in production for half a year at this point. Most requests are replied to in less than 1 ms. we anticipate that on a modest EC2 node, we should be able to scale to hundreds of thousands of users and field hundreds of requests per second. Should we need to, we can scale vertically 1-2 orders of magnitude beyond this point. Ultimately our backend plans to follow a scaling strategy similar to email where users have a home server. And our long-term vision is one of a network of decentralized server operators. But that's a dream that's still quite far away.

As a result, what Lockbook ultimately converged on, is probably my new default approach for building simple backend systems. If this intrigues you, check out the source code of db-rs or take it for a spin.

Currently db-rs exactly models the needs of Lockbook. there are key weaknesses around areas of concurrency and offloading seldom accessed data to disk. Whenever Lockbook or one of db-rs' users needs these things, they'll be added. Feel free to open an issue or pull request!

db-rs (attempt 2)

When Lockbook first began, it's architecture was I would consider the typical backend architecture. When a client would make a request, it would be load balanced to one of several api servers. The selected api server would communicate either Postgres, and S3 to fulfill that request. As we designed the product we understood our needs better and gradually re-evaluated various components of our architecture. We learned a lot through this process and although our needs are not unique, the architecture we're converging towards is unique in it's simplicity: a single api server, running a single Rust program. In part this architecture was enabled by an expressive, lightweight, embedded database I wrote called db-rs. Transitioning to this architecture allowed us to move much more quickly and we expect it to (conservatively) handle hundreds of requests a second made by hundreds of thousands of users all for around $50 / month. Today, for most typical projects, this would be my default approach.

Like most backends, we have users who have accounts, accounts manage our domain specific objects: their File. Both users and files have metadata associated with them, users have billing information, files have names, locations, collaboration information, size, etc.

We were finding that doing typical things in the context of a typical architecture was slow and annoying. For instance much of this data modeling and access would be done with SQL. Your backend however is certainly not written in SQL, so this requires some level of data conversion for every field that your server persists. There's likely to be subtle mismatches around how your langauge handles types (limits, signs, encoding) or even meta ideas around types (nullability). You may try an ORM which has it's own strengths and weaknesses. We also found that modeling certain ideas about files was hard to do at the database level: for instance, two files with the same parent cannot have the same name, unless one of them is deleted. Or even trickier: you cannot have a cycle in your file tree.

Maybe it's silly to try to do this in SQL, so instead you read all the relevant data into your application and run your validations in your server and only write to the database once you're satisfied with the state of your data. But make you're up-to speed on your transaction isolation types! Oh also make sure no one writes to your database without going through your server first otherwise they may invalidate your assumptions.

Okay maybe you do want to do this in SQL then, so you write a complicated query in a language with very little support for things like tests or package manager. And hope that you've expressed your query and setup your tables in a manner that performs acceptably. Okay let's say you've crossed all those hurdles, let's setup some environments: you can have your team install postgres directly and field complaints about it being a pretty heavy application or you can containerize it and field complaints about docker instead. In production you have to determine whether you need pg_bouncer for connection pooling. Okay what about configuring Postgres itself for production, can I just run Postgres on a Linux instance? Nope (from postgres.org):

PostgreSQL ships with a basic configuration tuned for wide compatibility rather than performance. Odds are good the default parameters are very undersized for your system.

Not too bad once you read through, but after some reflection this was the sort of thing slowing our team down. We had similar interactions with our load balancer and S3. In the past we've had similar intereactions with tools we've seen our day-jobs use at scale. We were ready to try something new to see if we'd have different outcomes. Our application code itself, while complicated, was a tiny fraction of the total request time. We re-architected to eliminate all network traffic from our server, instead of Postgres initially we used a mutex, a bunch of hash tables, and an append only log. Instead of S3 we saved files locally using EBS. We configured our warp rust server to directly handle tls connections rather than our load balancer. Our latencies across all our endpoints were down to less than 2ms without any attention paid to performance within our application layer. We realized we didn't need concurrency, or horizontal scalability just for it's own sake. We wanted our application to be able to scale to the userbase of our dreams, and bringing the latency of each endpoint down by several orders of magnitude was a far easier way to achieve that goal.

Inspired by the initial results I sat down to see how much progress I could do on the core idea. db-rs is what resulted. In db-rs you specify your database schema in Rust:

#![allow(unused)]
fn main() {
#[derive(Schema, Debug)]
pub struct CoreV3 {
    pub account: Single<Account>,
    pub last_synced: Single<i64>,
    pub root: Single<Uuid>,
    pub local_metadata: LookupTable<Uuid, SignedFile>,
    pub base_metadata: LookupTable<Uuid, SignedFile>,
    pub pub_key_lookup: LookupTable<Owner, String>,
    pub doc_events: List<DocEvent>,
}
}

A single source of truth, version controlled alongside all your other application logic. The types you see are your Rust types, as long as your type implements the ubiquitous Serialize Deserialize traits you won't have to write any conversion code to persist your information. You can select a table type with known and straightforward performance characteristics. Everything within the database is statically known. So all your normal rust-related tooling can easily answer questions like "what tables do I have", how do I append to this table? What key does this query expect? What value will it give me?

Moreover, when you query, you're handed references to data from within the database, resulting in the fastest possible reads. When you write, your data is serialized in the bincode format, an incredibly performant and compact representation of your data, persisted to an append-only-log, one of the fastest ways to persist a piece of information generally.

As a result of this new way of thinking about our backend, we don't have to learn the nitty gritty off:

• SQL
• Postgres at scale
• S3
• Load balancers

Locally using this database is just a matter of cargo run'ing your server, which is a massive boon for iteration speed and project onboarding. People trying to self host lockbook (not a use case fully supported just yet, but a priority for us) are going to have a significantly easier time doing so now.

If you're primarily storing things that could be stored within Postgres, and are writing a Rust application, the productivity and performance gains are likely going to be very similar for you. If you had a reference to all your data and could easily generate a response to a given API request within 1ms you're likely also looking at a throughput of hundreds of requests per second. If you're an experienced Rust developer think about how quickly you could get a twitter clone off the ground.

If this intrigues you, checkout the source code of db-rs or take it for a spin. The source code is less than 800 significant lines of code, and currently reflects the exact needs of Lockbook. It's very possible that it falls short for you in some way, for instance currently your entire dataset must fit in memory (like Redis), this is fine for Lockbook for the next year or so, but will one day no longer be okay. If this is a problem for you, feel free to open an issue or pull request!

Lockbook's architectural history (attempt 1)

Today Lockbook's architecture is relatively simple: we have a core library which is used by all clients to connect to a server. Both core and server are responsible for managing the metadata associated each file and it's content. Our server is a single mid-sized ec2 instance, and makes no network connections for file-related operations. Our core library communicates directly with our server. Operations that may be traditionally handled by a reverse proxy (ssl connection negotiation, load balancing, etc) are handled by a single rust binary. Our stack achieves throughput and scale by being minimal and fast: our server responds to all file related requests with sub-millisecond latency.

Our stack wasn't always this lean, when we first set out our stack looked much more traditional: we used haproxy to load balance requests and provide tls between 2 server nodes. Our server stored files in s3 and metadata in postgres. In core we stored our metadata in sqlite. For most teams, out growing a simple tool usually takes the form of adopting a more complicated-full-featured version of that tool. For us, outgrowing a tool often involved taking a step back and creating a simpler version of the tool that fit our needs better.

File contents

Take s3 for instance. We found that interacting with s3 was becoming too slow, and a source of, albiet rarely, outages. We saw 3 paths forward:

1. Invest deeper in s3. We could expose our users (encrypted) publicly, and have core directly fetch files from s3 instead of having our server abstract this away.
2. Make our architecture more complicated by caching s3 files somewhere.
3. Have our server manage the files itself, locally.

With s3 we had a handful of crates that we could choose between. If we managed files ourselves (writing locally to a drive), we'd be programming against a significantly more stable and well understood api: Posix System Calls. We could use ebs to make various tradeoffs for performance and cost. We would have a slight increase in code complexity as we'd need to learn how to do atomic writes (write the file somewhere temporarily, and atomically rename it when the write is complete). But we'd have a significant decrease in overall engineering complexity:

• no need to learn about s3 specific concepts (access control, presigned urls, etc).
• no need to simulate s3 in environments where using s3 is infeasible (local development, CI, on-prem deployments). No need to wonder if there's subtle differences between various s3 compliant products.
• significantly smaller surface area of failure.

File metadata

Initially file metadata was stored in Postgres, and to better understand why we moved away from Postgres, I should explain what our metadata storage goals are. When a user attempts to modify a file in some way we need to enforce a number of constraints. We need to make sure no one modifies someone else's files, no one updates a deleted file, no one puts a tree in an invalid state (cycles, path conflicts, etc), and so on. Initially we tried to express these operations and constraints in SQL and after a couple rounds of iteration it was clear this wasn't the right approach. Our SQL expressions were complicated, hard to maintain, and the source of many subtle bugs.

So we took a step back and moved significant amounts of our logic into rust. The flow of a request was now, a user is trying to perform operation X, fetch all relevant context from the database, perform the operation, run validations, persist the side-effects. This moved most of the complexity back into rust where we could easily write tests, use 3rd party libraries, and iterate quickly with a compiler.

Even with this refactor, we were still largely unsure about our usage of Postgres. Managing Postgres at scale is non-trivial, the surface area of learning how to configure postgres to keep more data in memory and juggle multiple parallel connections (pg-bouncer) is pretty large. Additionally the local development experience of Postgres was pretty poor, it either involved a deep install on your system, or nescesitated containers. And ultimately there were subtle differences between how it may be configured locally and in production, differences which could meaningfully impact the way queries executed. Finally we were willing to do more up-front thinking about how we would store and access data. We didn't need the flexibility of SQL, and found ourselves facing more problems due to the declarative nature of SQL.

Since most of the complicated parts were in Rust, switching to Redis was a fairly inexpensive engineering lift. It was significantly easier to reason about how Redis would behave in various situations and manage it at scale under load. Redis was dropped in as a replacement to Postgres, and with this replacement we were able to eliminate an organization wide dependency on docker. Another set of associated concepts we'd no longer need to reason about to achieve our goals. Our team experienced a vastly better local development experience from this change.

It was now time to pay attention to core. Core shares similar goals to our server with regards to the operations it's trying to perform, but it is additionally constrained by requiring an embedded database and is sensitive to things like startup time and resource requirements. Core also needs to be easy to build for any arbitary rust target, the ideal database would probably be a pure Rust project. Our journey started with SQLite and was a bumpy one initially for some of our compilation targets. But the journey ended the moment we were no longer interested in expressing complicated operations in SQL. Informed by our server-side experience, we left the problem intentionally unsolved for a while as we invested in other areas of the project. We simply persisted our data in a giant JSON files. As we expected while we were in the early days we experienced issues of data corruption as sometimes our writes would be interrupted or multiple processes sharing a data directory would cause data-race-conditions.

db-rs

After investing in other areas of our project I had done a lot of thinking around databases, especially around what would be ideal for a project that wanted to express as much in Rust as possible. I wanted a database that was fast by virtue of minimalism. For instance, simply being embedded affords your application a massive amount of throughput. For our project, this also meant that we could just stick our database behind a Mutex and significantly reduce the number of problems we're trying to solve at the moment. I wanted a database that was designed with rust users in mind and ultra-low-latency. I also wanted to provide rust users with an ergonomic way for users to express a schema, with rust types (not database specific types) and not have them worry about serialization formats.

We needed a database that was:

• embedded
• fast
• ergonomic
• durable

So in about a week I created db-rs.

Once db-rs existed, with the abstractions present in core and server, it was easy to drop it in. Once again, this simplification boosted performance massively, simplified our code, and simplified our infrastructure, and reduced the number of foreign concepts that our team needed to understand and fbuild around.

With the request latency the lowest we'd ever seen it, without any significant effort to optimize our code (just eliminate things we didn't want), we also eliminated nginx and just had warp perform tls handshakes and commit to a single server node for the near future. We estimate this modestly priced ec2 instance ($50 / month) can handle hundreds of requests a second from hundreds of thousands of users. If we need to, we have a healthy amount of vertical scaling headroom. Beyond that, our long term plan involves a scaling strategy similar to what's used by self-hosted email.

Today our only remaining project dependency for most work is just the rust toolchain. Local dev environments spin up instantly without the need for any container or network configuration. Deploying a server means building a binary for linux and executing it.

Lockbook's Editor

Building a complex UI element in Rust that can be embedded in any UI framework

As Lockbook's infrastructure stabilized, we began to focus on our markdown editing experience. Across our various platforms, we've experimented with a few approaches for providing our users with an ergonomic way to edit their files. On Android, we use Markwon, a community-built markdown editor. On Apple, we initially did the same thing but found that the community components didn't have many of the features our users were asking for. So as a next step, we dove into Apple's TextKit API to begin work on a more ambitious editor.

Initially, this was fine, but as we worked through our backlog of requests, I found things that were going to be very time-expensive to implement using this API. We were having performance problems when editing large documents. The API was difficult to work with, especially because there were no open existing bodies of work that implemented features like automatic insertion of text (when adding to a list), support for non-text-characters (bullets, checkboxes, inline images), or multiple cursors (real-time collaboration or power user text editing). Even if we did invest the effort to pioneer these features using TextKit, we would have to replicate our efforts on our other platforms. And lastly, none of my other teammates knew the TextKit API intimately, so I wouldn't be able to easily call on their help for one of the most important aspects of our product. We needed a different approach.

In the past, I've discussed our core library -- a place we've been able to solve some of our hardest "backend" problems and bring them to foreign programming environments. We needed something like this for a UI component we needed a place where we could invest the time, build an editor from the ground up, and embed it in any UI library.

We considered creating a web component. Perhaps we could mitigate some of the downsides of web apps if we were only presenting a web-based view when a document was loaded. Maybe we could leverage Rust's great support for web assembly for the complicated internals. Ultimately I felt like we could do better, so I continued thinking about the problem. On Linux, we'd begun experimenting with egui: a lightweight, Rust, UI library. Their README had a long list of places you could embed egui, and I wondered if I could add SwiftUI or Android to that list.

And so began my journey of gaining a deeper understanding of wgpu, immediate mode UIs, and how this editor might work on mobile devices.

Most UI frameworks have an API for directly interfacing with a lower-level graphics API. In SwiftUI, for instance, you can create an MTKView which gives you access to MetalKit (Apple's hardware accelerated graphics API). Using this view, you can effectively pass a reference to the GPU into Rust code and initialize an egui component. In the host UI framework you can capture whichever events you need (keyboard & mouse events for instance) and pass them to the embedded UI framework. It's the simplicity of immediate mode programming which enables this to be achievable in a short period, and it's the flexibility of immediate mode programming which makes it a great choice for complex and ambitious UI components. The approach seemed like it held promise so we gave it a go.

After a month of prototyping and pair programming with my co-founder Travis, we had done it. We shipped a version of our Text Editor on macOS, Windows, and Linux which supported many of the features our team and users had been craving. The editor was incredibly high-performance, easily achieving 120fps on massive documents. Most importantly we have a clear picture of how we would go about implementing our most ambitious features over the next couple of years.

After we released the editor on the desktop, we began the process of bringing it to mobile devices. This was a new frontier for this approach. On macOS, we just had to pass through keyboard and mouse events. On a mobile device, there are many subtle ways people can edit documents. There are auto-correct, speech-to-text, and clever ways to navigate documents. After some research, we found a neatly documented protocol -- UITextInput -- which outlines the various ways in which you can interact with a software keyboard on iOS. We also found a corresponding document in Android's documentation.

So back to work we went. We expanded on our SwiftUI <--> egui integration giving it the ability to handle events that egui doesn't recognize. We piped through these new events, refined the way we handle mouse/touch inputs, and a couple of weeks ago, we merged our iOS editor bringing many of our gains to a new form factor.

We're very excited about the possibilities this technique opens up for us. It allows us to maintain the look & feel that users crave while giving us an escape hatch down into our preferred programming environment when we need it. Once our editor is more mature and the kinks of our integration are worked out, we plan to apply this strategy to more document types. Long term we're interested in making it easy for people to quickly spin up their own SwiftUI component backed by Rust (as presently this still requires a lot of boilerplate code).

On net, the editor has been a big step forward for us. It's already live on desktops and will be shipping on iOS as part of our upcoming 0.7.5 release. It's a large and fresh body of work, so we anticipate some bugs. If you encounter any, please report them to our Github issues. And, as always, if you'd like to join our community, we'd love to have you on our Discord server.

Defect hunting

When designing Lockbook we knew we wanted to support a great offline experience. To our surprise, this grew to become one of the largest areas of complexity. Forming consensus is an active area of research in computer science, but Lockbook has an additional constraint. Unlike our competition, large areas of complexity take place on our user’s devices that can't update remotely. Additionally, the administrative action we can take is limited: most data entered by users is encrypted, and their devices will reject changes that aren’t signed by people they trust. All this is to say that the cost of error is higher for our team and it’ll likely take longer for our software to mature and reach stability. Today I’d like to share a tester we created to help us find defects and accelerate the maturation process. We affectionately called this tester “the fuzzer”. We’ll explore whether this is a good name a bit later, but first, let’s talk about the sorts of problems we’re trying to detect.

Users should be able to do mostly anything offline, so what happens if, say Alice moves a document and Bob renames that document while they were both offline? What happens if they both move a folder? Both edit the same document? What if that document isn’t plain text? What if Alice moves folder A into B, and Bob moves folder B into A?

Some of these things can be resolved seamlessly while others may require user intervention. Some of the resolution strategies are complicated and error-prone. The fuzzer ensures regardless of what steps a user (or their device) takes various components of our architecture always remain in a sensible state. Let me share some examples:

• Regardless of who moved what files and when, we want to make sure that your file trees never have any cycles.
• No folder should have two files with the same name. Creating files, renaming files, and moving files could cause two files in a given location to share a name.
• Actions that change a file's path or sharees could change how our cryptography algorithms search for a file's decryption key, we want to make sure for the total domain of actions your files are always decryptable by you.

As we used our platform we've collected many such validations that we want to ensure never occur for the global set of actions on our platform, and the fuzzer's job is to spit out test cases that violate these constraints. It does this by enumerating all the (significant) actions a user can take on the platform across N devices and with M collaborators. It randomly selects from this space and at each step it asks all parts of our system to make sure everything is still as it should be. It does this process in parallel fully utilizing a given machine's parallel computational resources. It travels through the search space in a manner that limits the amount of recomputation of known good states, fully utilizing a given machine's memory.

Generally when people say they're fuzzing, they mean handing a given function or user input randomized input to try to produce a failure. Our fuzzer captures that spirit of this but is different enough that plenty of people have raised an eyebrow when I told them we call this process fuzzing. Unfortunately test simulator isn't as cool a name, knowing what this process is now, if you can think of a cool name, do tell us.

Today our highly optimized fuzzer executes close to 10,000 of these trials per second, however initially it was just a quick experiment I threw together to gain confidence in an early version of sync. This was written when we were still using an architecture that used docker-compose to spin up nginx, postgres, and pgbouncer locally. The fuzzer almost immediately found some bugs. We fixed these bugs, and the value of the fuzzer was made apparent to us. The time between defects started to grow and so did the intricacies of the bugs the fuzzer revealed. As a background task, we continued to invest in the implementation of the fuzzer, and the hardware it ran on. As our architecture became faster, so did the fuzzer alongside it. Today the fuzzer has been running continuously for months and has verified 10s of billions of user actions, a promising sign as we get ready to begin marketing our product.

Below are some of the fuzzer's key milestones. If you're interested in browsing the most recent implementation you can find it linked here.

15 Trials Per Second

Initially, we were running the fuzzer on our development machines, kicking it off overnight after any large change. Our first big jump in performance came from deciding to run it on a dedicated machine and trying to fully utilize that machine's computational resources.

80 Trials Per Second

We first tried running our fuzzer on a dedicated server which had 80 vcpus. We purchased this machine for $600 in 2020. Most of our early optimization efforts centered around tuning Postgres to perform better.

250 Trials Per Second

Our next largest jump in performance was when we made the switch from Postgres to Redis, and upgraded the hardware that the fuzzer runs on. After 2 years of faithful service, our Poweredge experienced a hardware failure which we weren't motivated enough to diagnose. So in 2022 we pooled our resources for a 3990X Threadripper with 120 vCPUs.

900 Trials Per Second

Before [db-rs] there was [hmdb] which was similar in values to db-rs, but worse in execution. It still served our needs better than Redis and performed better as it was embedded in the process rather than something that communicated over the network. It additionally used a vastly more performant serialization protocol across the whole stack, inside [core] and our server.

4000 Trials Per Second

In late 2022 I wrote created a compile-time flag in [core] which allowed us to directly compile the entire server into core during test mode. This meant that instead of executing network calls for fetching documents and updates core was directly calling the corresponding server functions. At this point, no part of our test harness was using the network stack

10,000 Trials per second

Once db-rs was fully integrated into core and server, I added a feature to db-rs called no-io, which allowed core and server to enter a "volatile" mode for testing. This also allowed instances of core and server, and their corresponding databases to be deep copied. So when a trial ran, if most of the trial had been executed by another worker, it would deep copy that trial's state and pick up where it left off.

Future of the fuzzer

Personally, the fuzzer has been one of the most interesting pieces of software I've worked on. If, like me, this piques your interest and you're interested in researching ways to make it faster with us, join our discord.

Creating a SICK CLI

At Lockbook we strongly believe in dogfooding. So we knew alongside a great, native, markdown editing experience we would want a sick CLI. Having a sick CLI creates interesting opportunities for a niche type of user who is familiar with a terminal environment:

• They can use the text editor they're deeply familiar with.
• They can write scripts against their Lockbook.
• They can vastly reduce the surface area of attack.
• Can always maintain remote access to their Lockbook via SSH.

In this post I’m going to tackle 3 topics:

1. What makes a CLI sick?
2. How do you go about realizing some of those “interesting opportunities” using our CLI?
3. What’s next for our CLI?

What makes a CLI sick?

It’s tab completions. For me, tab completions are what I use to initially explore what a CLI can do. Later, if the CLI is sick, I use tab completions to speed up my workflow. I don’t just want to tab complete the structure of the CLI (subcommands and flags). I want to tab complete dynamic values, in Lockbook's case, this means completing file paths and IDs.

If you're creating a CLI most libraries make you choose between a few bad options:

• Hand-craft completion files for each shell.
• Sacrifice dynamic completions and just settle for automatically generated static completions.

Rust is no exception here, clap has some support for static completions, but no way to invoke dynamic completions without writing a completion file for each shell.

And so we set out to solve this problem for the Rust ecosystem, and created cli-rs. A parsing library, similar to clap but with explicit design priorities around creating a great tab completion experience. As soon as cli-rs was stable enough we re-wrote lockbook's CLI using it so we could pass on these gains to our users.

Cli-rs is simple, you describe your CLI like this:

#![allow(unused)]
fn main() {
Command::name("lockbook")
    .description("The private, polished note-taking platform.") 
    .version(env!("CARGO_PKG_VERSION"))
    .subcommand(
        Command::name("delete")
            .description("delete a file")
            .input(Flag::bool("force"))
            .input(Arg::<FileInput>::name("target").description("path of id of file to delete")
                        .completor(|prompt| input::file_completor(core, prompt, None)))
            .handler(|force, target| delete(core, force.get(), target.get()))
    )
    .subcommand(
        Command::name("edit")
            .description("edit a document")
            .input(edit::editor_flag())
            .input(Arg::<FileInput>::name("target").description("path or id of file to edit")
                        .completor(|prompt| input::file_completor(core, prompt, None)))
            .handler(|editor, target| edit::edit(core, editor.get(), target.get()))
    )
...
}

This gives the parser all the information it needs to offer the best tab completion behavior. It handles all the static completions internally and then invokes your program when it’s time to dynamically populate a field with your user’s data.

We also invested a ton of effort in the infrastructure that deploys our CLI to our customer's machines so that tab completions would be set up for most people by default.

Exciting Opportunities for Power Users

Use your favorite text editor

You can lockbook edit any path you have access to and our CLI will invoke vim, utilizing any custom .vimrc that may exist. You can override the selected editor by setting the LOCKBOOK_EDITOR env var or using the --editor flag. So far we support vim, nvim, subl, code, emacs and nano.

If we don’t support your favorite editor, send us a PR or hop in our discord and tell us.

Extending Lockbook

We want Lockbook to be maximally extensible, this extensibility will take many forms, one of which is our CLI. Let's explore some of the interesting things you can accomplish with our CLI.

Let’s say you wanted a snapshot of everything in your second brain decrypted and without any proprietary format for tin-foil-hat backup reasons. You can easily set a cron that will simply lockbook sync and lockbook backup however often you want. lockbook export can be used to write any folder or document from Lockbook to your file system, paving the way for automated updates of a blog. Edit a note on your phone, and see the change live on your blog in seconds. lockbook import lets you do the opposite. Want to continuously back up a folder from your computer to Lockbook? Setup a cron that will simply Lockbook import and then lockbook sync.

Ultra secure

I like to think about security as the product of a few numbers. So if, for example, you’re product is closed source, one of those numbers in your multiplication chain is a big fat zero. And there’s nothing you can do to pretend it’s secure. Similarly, the age of a product is one of those numbers. Newer is worse, and this is one of Lockbook’s current weaknesses.

But one of Lockbook’s strengths is how much you can reduce the total amount of code it takes to interact with Lockbook. On one end of the spectrum, you have software that requires a full browser installation to perform the most basic tasks. Slightly better than that is software that runs natively, and on the other end of the spectrum is software that doesn’t even rely on a UI library. Once we’re mature, if you wanted to run Lockbook on a libre-booted thinkpad running an ultra-minimal operating system, Lockbook wouldn’t require you to add the Google Chrome dependency tree to your setup.

Remote Lockbook

Sometimes you find yourself employed by a financial institution that heavily restricts what you can do on their machines. Without thinking too much more about your situation you may want to simply add something to your grocery list without pulling out your phone. Unfortunately, IT has locked down your remote Windows 7 installation, and not only can you not install our Windows app (which does not require administrator privileges to install) but you cannot visit GitHub itself!

Maybe in this environment, it’s not worth it to update your grocery list, but you identify with the likes of Ron Swanson, and you will not be defeated by your IT department. How? Because you port forwarded your desktop and memorized a lengthy SSH password. So you ssh in, use your favorite text editor, and you update that grocery list. There’s no stopping you.

What’s next for our CLI

Our CLI has come a long way, we've experimented with various ways of allowing you to quickly find a note and edit it. In the past we experimented with piping output to programs like fzf, we even tried implementing a custom full-screen search. This is the approach that feels the best to us and we think is going to stand the test of time. But work is never done, so here are some of the things we plan to tackle in our CLI:

• Continue to invest in our release infrastructure to bring our CLI to more package managers. If you'd like to become a maintainer for a particular distro reach out!.
• Support richer parser inputs including variable number of arguments, grouped command line flags, and logical de-duplication of tab completions (this flag or argument is already specified so don't suggest it again).
• Deeper integrations with shells in cli-rs: offer ways to express that this argument is a normal file with completions, or implement mechanisms to re-write the current prompt (lockbook edit sick<tab> tab completes: lockbook edit writing/parth.cafe/creating-a-sick-cli.md presently tab completion options must begin with the current prompt).
• A richer showcase of interesting things we can do with our CLI, we plan to set up our blog the way I described above and provide concrete examples of how to do many of the things I outlined. So if you haven't already subscribe to the Lockbook Blog, and Lockbook Youtube Channel.

Multimedia updates!

In a few videos we've expressed that we've been focusing on building up our infrastructure to better support multimedia and increase overall platform stability. Today I'd like to share some exciting updates on the multimedia front!

Lockbook Workspace

In a previous post, we shared details about our cross-platform markdown editor, which allowed us to increase the complexity and interactivity of our editor. After working out the kinks with the initial implementation we've doubled down on this strategy and expanded it to the entire tab strip and all content displayed within Lockbook. We call this component the Lockbook Workspace. As a portion of the team brought workspace to all of the platforms, Adam redesigned our drawing experience from the ground up to use SVGs instead of a proprietary drawing format. Canvas deserves its own post, so stay tuned for that. In addition to SVGs and Markdown, workspace brought image and PDF previews to all platforms.

Markdown and SVG have crystalized as the 2 main document formats that will be editable inside Lockbook. These open formats are a natural fit for our customer-centric values. They both offer broad compatibility across the internet and don't lock you into our platform. The freeform expression of SVGs complements the restricted feature set of Markdown nicely. SVGs are also trivially embedded inside markdown documents as images.

Supporting these two document types offers our platform access to two very diverse types of note-takers. On one hand, there's the sort of people who like apps like OneNote, Notability, and GoodNotes for their unstructured creative surface. On the other hand, some people like the structure of apps like Notion, Obsidian, and Google Docs for their searchable, linkable, and collaborative experience.

A key idea of our platform is to not split your life among different apps all fundamentally storing bytes for you and your team. My co-founder Travis and are members of both of these modalities. Users like my Dad are heavy OneNote users, while there are plenty of users who are pretty firmly in the "strongly typed" category of markdown.

Lockbook Filesystem

When we were making early design decisions for Lockbook's system architecture, we considered the many ways we could allow our users to organize their thoughts. We considered a tag-like system similar to Bear that optimizes for flexibility (a note could live in two folders at once). We also considered a Google Keep-like single-note experience. But I strongly advocated for a "Files and Folders" hierarchy. I knew I wanted Lockbook to integrate deep into the traditional computing experience seamlessly. There are times when you want to edit Markdown using our fancy Markdown editor. And there are times when you want to edit spreadsheets and photos and have all the durability and security guarantees of Lockbook. Again the key idea here is to not split your life across various apps, we don't want you to have to use Google Drive / DropBox for one type of content and Lockbook for another.

As Lockbook emerges out of its state of running an ungodly amount of experiments, this was another area I wanted to de-risk and see how our platform and infrastructure performed. Tax season was also approaching and we had similarly flavored requests from people who wanted to use the types of apps we're never going to be in the business of creating.

We somewhat supported the basic workflow of dropping files in and out of Lockbook. But this is clunky and our users expect more from our ragtag crew of part-time developers. So I took a long weekend and tried to determine if we could do better.

The first thing that came to mind was to use the FUSE protocol. This would allow our users to seamlessly mount their Lockbook as if it's a flash drive on their computer. Any requests for bytes would be fielded directly by the Lockbook instance running on your computer and we don't need to be in the business of watching directories for changes and trying to reconcile changes. This was our target UX but unfortunately, FUSE only works well on Linux, and on macOS users would have to install some pretty invasive 3rd party software.

A close second flavor of the same thing is the NFS protocol. Offering a similar experience to FUSE, with some additional baggage associated with the network. Fortunately, this works pretty seamlessly on both macOS and Linux. I stumbled upon an NFS-Server crate which made prototyping a very productive experience.

In a couple of days, I had a high-performance implementation ready. Mounting my whole Lockbook directory and seeing it work effortlessly with Lightroom, Keynote, and CAD software was a pretty magical moment for me as an engineer.

We shipped an early preview of lb_fs in our CLI in 0.9.0. So far we've experienced some good feedback. We even had one of our users use the filesystem as part of an automation pipeline for their dotfiles!

We're excited about the future of lb-fs. We intend to integrate it directly into our desktop apps -- allowing you to click on any document not supported by Workspace and open it natively on your computer. We also want you to be able to click any folder and "Open Externally". Allowing you to seamlessly backup the decrypted contents as easily as copying them to a different location on your computer.

But at the moment we're pausing for some reflection: should lbfs continue investing in NFS? On Windows, it requires Windows Pro (which most users don't have, and there is no 3rd party stopgap). Potentially, we could seek a higher quality platform-specific interface like ProjFS. If we were to explore something like that on macOS we could add nice touches like showing the sync status or collaboration details within Finder itself.

This could also be a cool opportunity to create a state-of-the-art, cross-platform virtual file system abstraction for the Rust ecosystem. If you'd be interested in pursuing something like that please join our discord and reach out! We'd be happy to support you in any way we can.

Other Infrastructural updates

Further investments in multimedia right now are bottlenecked by our current networking implementation. We expect it to be a small lift to unlock the potential of ws and fs by using a slightly more sophisticated approach to networking:

• don't sync all your files all the time -- don't sync large files to my iPhone, let me log in immediately and lazily fetch files as needed (still have a well-managed cache for great offline support).
• be able to more reliably sync large files -- presently, especially under adverse network circumstances, large files are particularly problematic.
• fetch and push files in parallel.

These features and a few others comprise our Sync 4.0 tracker, the product of which has significant implications for everything mentioned above. Sync 4.0 should also allow us to sync way more aggressively (where appropriate) -- a long-standing request from almost every type of user.

Once we have this situation under control we can start to look toward a future where we have a richer set of tools for collaboration:

• document revisions
• document comments
• author history (like git blame)
• dare I say -- Google Docs style real-time collaborative editing?

These features will likely be presented in workspace itself, but will be present for all file types fundamentally.

You can track the broader multimedia efforts here.

If you have a specific use case that we didn't cover above we'd love to hear from you! Join our discord and share your thoughts!

Footnote: Community Document Types

Another interesting opportunity that workspace presents our community is the ability to author custom new document types. If you can come up with a data format, and write an egui widget then you can contribute a new file type to Lockbook workspace.

Presently we have some community members pursuing interesting visualizations of disk space and links within markdown documents inspired by Disk Usage Analyzer on Linux and the Obsidian Graph View.

We've also heard lots of interesting ideas about building habit trackers and to-do lists this way as well. If you're interested in whipping up something like this join our discord! For now, we will play an active role in what file types make it to all users, but one day this structure may grow into a formal plugin system!

What does secure mean?

We call Lockbook a secure product. This isn't a unique claim to make as you can see below.

.

So what's the difference between Notion's claim and ours? It's simple, we can't see your notes even if we wanted to, and Notion can. There may be policies, protections, and certifications to ensure they don't do anything improper with your data, but they have the ability to see your content. In contrast, we don't have that ability.

Before your content departs from your device it's encrypted in such a way that only you or any collaborators you chose can decrypt the data. Our server never receives the decryption keys so we can't spy on you, our hosting provider can't spy on you and the government can't spy on you. This is called end-to-end encryption. You can learn more about this here.

How do I know I can trust Lockbook?

Because we don't believe in security through obscurity, all of our code is open source. We want to make it as easy as possible for someone to audit our code. All of the code that would be relevant to an audit is written in a single language and relies on well-established cryptographic primitives like eliptic curve cryptography and AES.

The keen-eyed among us ask: how do we know you're running and shipping the code that's on GitHub? You don't, but we engineered lockbook so that the app running on your phone (the client) doesn't trust our server either. It doesn't send any secrets and verifies (cryptographically) the information it receives.

To be frank -- we don't trust our server either! We consider our cloud infrastructure to be an adversarial environment and have engineered the whole product to be resilient to snooping and unexpected downtime. This is part of the reason we placed such a great emphasis on offline support. Long term we envision a decentralized future (much like the email protocol), but that's a daydream for the time being.

For your convenience, we build and publish our apps to a variety of marketplaces of varying trustworthiness (like Apple's App Store). But we've also made it as easy as possible for you to build any of our apps directly from source to cut out all the middlemen -- allowing you to be absolutely certainty that you're running the code you expect.

Why should I care if Google can see my content?

Some people are shocked to find out that most companies can just see all your content.

Others don't really care if some select group of people have access to their documents. They don't fear the government because they don't think they are or ever will be a target (I hope they're right).

Despite this, they still don't want their friends, families, or enemies to see their content. It's private and they'd like to keep it that way.

Companies, however, get hacked all the time. They leak passwords emails, and content all the time. The perpetrator could be anyone from a foreign government to a talented prepubescent basement dweller. It could be a disgruntled employee or a misconfigured database. When the data sitting on the server is compromised do you want it to be a garbled mess of encrypted data, or do you want it to be every photo you've taken in the last 15 years?

So is my Lockbook unhackable?

A useful mental model for determining how secure you are is something like "How much would it cost to compromise me?".

To compromise most traditional companies the exploit may be as simple as waiting for an employee to make a mistake and expose secrets. If that doesn't work an attacker could explore bribing an employee or even infiltrating the company.

Because the data sitting on our server is encrypted, for someone to gain access to your content they have to compromise your individual device. Hacking into an individual's device is several orders of magnitude more expensive. Apple is willing to pay up to $2M bounties for certain types of device compromises. Widescale compromises of devices like the iPhone are far more rare than the compromises of online services.

Often with these types of compromise physical access is required. So if someone wants to get your content they may need to hire a thief to break into your home, and then use an exploit that may be worth millions of dollars.

Additionally, Lockbook goes to some lengths to make sure you don't compromise yourself. We generate a key for you with 256 bits of entropy (a very long unguessable password). This means you can't accidentally re-use a password and compromise yourself.

This is an irrecoverable key eliminating the chance of an email or phone compromise additionally compromising your Lockbook.

You're not, however uncompromisable. There is a strong element of personal responsibility when it comes to security. If you're out there downloading random cracked-photoshop.exes much of your protection goes out the window.

Security is a chain, and attackers will seek to exploit the weakest (cheapest) link in your setup.

How do I use Lockbook Securely?

The design of Lockbook makes your content as secure as your devices are. Here are some very broad general recommendations for how I think about security.

Mobile Devices

Mobile devices are generally inherently secure devices. Apps are running in a very limited execution environment and can't access each other's data. Use popular devices, keep them fresher for ~4 years, update your software, and use the security features your phone ships with (1111 is not a good passcode). If you do all this, you're at a pretty strong starting point.

As I mentioned above it takes a fair amount of sophistication for someone to break into an iPhone that's up to date. Your local police department can't do it, and probably not all of the 3 letter federal agencies can (but a few probably can). I think it's probably best to consider big tech and government synonymous from an adversarial perspective.

Google has some incredibly strong incentives to spy on you. They don't make the slightest effort to implement end-to-end encryption in their messengers and often they send computations to a server that Apple performs locally. Apple is incentivized to sell devices not ads, Apple has formed a reputation for simplicity and security.

Apple rules the app store supply chain with an iron fist which introduces attack vectors for supply chain attacks and censorship.

Both iOS and Android have open-source roots but neither of these platforms are practically open source. Both platforms have huge amounts of unremovable infrastructure that is closed source.

There are, however, secure phone implementations like Graphene, Librem, Liberty, and the Pinephone. The theme is generally: reasonably secure hardware and fully open-source software. Configured by default to make it easier to use securely and privately.

There is a school of thought that says phones are unsecurable. It's just too hard to have a cell plan that isn't coarsely tracking you with cell tower metadata. This way of thinking also considers phones to be toxic machines of addiction and control and recommends using dumb phones or no phones at all.

Computers

The surface area of attack on computers is far broader and we depend on computers for essential activities.

The security situation on Windows is a bit of a disaster, many Windows computers come pre-packaged with a large amount of bloatware. This increases the surface area of attack for devices. Windows is a fragmented ecosystem so suggestions to "only install sandboxed apps from the Microsoft Store" are effectively impractical. The normal way to install and update things is to download an exe from a website, a pretty bad security default.

In my opinion, Macbooks are a step up in security from Windows. Things on macOS are closer to iPhones, by default, the installation comes with minimal bloat and the security defaults are reasonable. Apps have to ask permission to access various resources. Additionally, you can turn off some security features and download apps from arbitrary locations. However, both Windows and macOS are largely closed source leaving room for "bugs" and backdoors.

The next step up here would be to run an open-source Linux flavor on commonly available hardware (Thinkpads, Dell laptops, or a Desktop computer). "Distributions" like Ubuntu, Fedora, and Manjaro provide a gentle introduction to Linux. These days the transition may be easier than you'd expect as most computing can happen in the browser.

This is a pretty solid place to arrive. Most of what you're running is open source and inherently more secure. You'll likely install and update all your software through your distro's package manager, either graphically or through the command line.

From here our next stop is minimal, hand-crafted Linux distros. On distros like Arch, and Gentoo you gain an understanding of what all the moving pieces of your computer are, and you play an active role in their assembly. Once you're setup you have a strong understanding of everything that's running on your system. What's running on your system is likely just what you need, and nothing you don't. This means your surface area for attack is as small as possible. At this step, you're probably already familiar with command line interfaces, and can likely use the Lockbook CLI rather than the desktop version -- further reducing the surface area of attack.

At this stage you may also consider exploring the BSD variants (historically more secure kernel than Linux), and specialized operating systems like QubesOS / Tails. You may also consider specialized hardware that seeks to remove eyebrow-raising hardware components like the Intel Management Engine.

Physical Security

It's also worth noting that there are very few software solutions to the wrench attack:

If your secrets are valuable enough to guard, consider investing in some real physical security. Understand that your adversaries may be immoral actors who will happily kidnap your family members to make you do what they want.

Is Lockbook secure enough for me?

Lockbook is a young piece of software, and as such may contain bugs. We believe we're secure by design, we've designed everything around making sure that you'll never leak information because of a bug. Rather our worst bugs should be crashes, missing documents, or the inability to communicate with our server. We do believe we're in a fundamentally different category of security than products that aren't end-to-end encrypted and open source and you are better off using us compared to those solutions.

But if your life literally depends on the security of your technology there's no avoiding dramatically investing a thorough knowledge of security and evaluating your chosen solution (lockbook, signal, PGP, local-only solutions, literally not using technology) at a very deep and fundamental level.

How do I learn more about security?

Here are some resources I've found valuable for better understanding the world of security:

• YT: MentalOutlaw
  • OpSec news
  • Analysis of how various people have been compromised
  • Instruction for higher security setups
• Podcast: Darknet Diaries
  • Interviews of hackers, spies, and an exploration of hacking culture
• Blog: Jameson Lopp
  • Crypto-focused OpSec discussion
• Blog: bigtech.fail
  • "Shining a light on the censorship, propaganda, and mass surveillance from today's tech corporations and governments."
• YT: Low Level
  • Technical analysis of high-profile security vulnerabilities

future race car

I’ve enjoyed a number of exciting vehicles so far. I’ve owned some motorcycles, off-road vehicles, and performance-oriented cars. I’ve modified most of these and I’ve been to the track a handful of times. Both of these things made me curious about building a dedicated track car. I’ve particularly been drawn to cars with such a rich aftermarket that they’re called platforms.

So in my exploration, I considered a number of platforms, I have a bunch of experience with Audi, but I wanted a more traditional rear-wheel drive, manual sports car. Audi’s are generally AWD cars, and manual variants are rarer due to a pretty good dual-clutch transmission. BMW seemed like a good iteration, I could get something newer and treat it as a normal second car for a few years and then turn it into a track car later. Or I could get something older and begin building immediately. I ended up shying away from this option for a few reasons.

• Luxury German cars start their lives as a racecar much heavier. This means you’re going to run through consumables like brakes, and tires much more quickly. And you have to make more power to be competitive.
• You’re paying a tax for the brand name at every interaction. Performance upgrades cost more when you want to go faster, and replacement parts cost more when you go too fast and break things.
• You’re more often fighting the manufacturer who wants you to bring the car into a dealership. When you’re changing rear brakes on vehicles that have electronic parking brakes you sometimes have to put the car into a service mode. On a Ford vehicle, this involves pressing and holding a couple of buttons while you start the car. On Audis this procedure can only be done either through a dealer or by using a grassroots OBD2 bluetooth device and app which costs a micro-transaction for a variety of such services. There are also generally more luxuries in your way which the removal of may not be handled particularly gracefully by the car (racing seats and steering wheel come to mind).

After mulling this over I knew I was either choosing an American pony car (Mustang / Camaro / Corvette) or a Japanese Tuner (Miata, Z, BRZ, Supra). This was a tough choice as I’m a die-hard American, and I presently have an F-150 and surely some of that mechanical familiarity would transfer over to a Mustang. Chevy LS swaps are also pretty sick. But ultimately I opted for the foreign import for the lower cost of entry. I was particularly blown away by what I saw surrounding the Mazda Miata.

Depending on your criteria you can probably get into a Miata for as low as $4k. To most Miata grey beards this is an all-time high price, it’s not uncommon to hear about someone getting a basic Miata for $1k a couple of years ago. This leaves plenty in the build department. This also leaves plenty in the replace-the-car-if-I-destroy-it department.

And boy can you build. Anything you saw them do in a Fast and Furious movie you can do to a Miata. Almost every part is widely available and inexpensive from a restoration perspective. There are a number of companies dedicated to providing specifically the Miata with aftermarket parts like Flyin’ Miata. There’s a rich knowledge base in forums and on YouTube. Donut Media (one of the largest car YouTube channels) did a series of videos building a track Miata. There are a number of performance strategies being explored including:

• Turbocharging
• Supercharged Miatas
• Swapping engines to the reliable and more powerful Honda K-Series motor
• Swapping engines to the monstrously powerful LS V8
• Or dropping ~35% of the weight of the Miata to create the Exocet

The Miata is a “generation” based car, like the Jeep. In its lifespan since 1990 there have been 4 generations of Miata: NA (‘90-’97), NB (‘98-’05), NC (‘05-’15), ND (‘15-). They all have a very similar personality:

• • Very light
• Great handling
• Not that powerful

The price gets rather high in the NC & ND range and I’m not sure it’s as compelling a purchase. The NA and NB Miatas however, are maybe the best value in the niche I’ve described so far. A great value means more room in the budget for building (and replacing the car if I go into a wall at 110mph).

There is however a key distinction between the NA and NB.

The NA Miata has pop-up headlights which make the car cute and charming.

The NB Does not.

So NA Miata it was, and NA’s are a little rare in my area. But even within the NA Miata there’s a faction. During ‘90-’93 the Miata used a 1.6L engine, ‘93 onwards they used a 1.8L engine in the US (~10% more powerful). I found that this engine has a ~$1k premium. The second performance bit I was looking for was a Torsen Limited Slip Differential. A differential helps your drive wheels rotate at different speeds in turns. In traction-limited settings (off-road or while shredding tires) a normal differential will allow the power to get sucked into the wheel that loses traction, which guarantees that wheel will not regain traction. An LSD helps deliver power to both wheels regardless of traction levels. In my search, an LSD inflated the price of the car another $1k.

And so, for $8750 I picked up a British Racing Green, M-Edition Miata from ‘97 with ~113k miles. The color is rare and is also my wife’s favorite color. Doing a test drive, finding no rust, and seeing above-average service history sealed the deal for me.

Worry not, I’ll update this picture with one that doesn’t include a dumpster full of construction debris when I get a chance.

I’m excited to use this blog to share the journey this Miata will take as it is restored, modernized, and upgraded for the race track.

Taking Stock of Our Purchase

As a recap, I picked up this '97 Miata a couple of weeks ago.

Now that it's in the garage, it's time to take stock of our new toy. So the car has 113,000 miles at the moment. In the manual, the largest maintainence interval is 60k miles, which means anything that I'm not absolutely sure was done, needs to be done now. So what do we know about this car so far?

I mentioned I was pleased with the vehicle history report. To summarize, it said:

• The car has had about 4 prior owners
• It's been owned by people who serviced it at dealerships throughout its life. This means it's bone stock, and pretty well maintained. I'm surprised about the couple of items I don't see on this list, but we'll get into that later.
• Apart from the oil changes and various checking of parts, the carfax report shows the following items have been replaced:
  • water pump
  • radiator hoses
  • camshaft position sensor
  • distributor seal
  • clutch slave cylinder
  • radiator clamps
  • thermostat, housing, and gasket

Nothing alarming there. Among other things, absent from this history report are transmission and differential fluid change. That's pretty easy on these cars, there's a fill bolt and a drain bolt. The fill bolt is at a location that constrains the maximum amount of fluid, so you drain the old fluid and pump in the new fluid until you see it overflow.

Also absent from the history is any record of spark plugs being replaced. Easy enough:

These were way past due, you shouldn't be able to tell the gap difference with the naked eye. Also replaced the plug wires while I was at it, kinda cool to see a company make the same product for over 30 years.

While I had the car jacked up, I saw that one of my fender liners was held up by thoughts and prayers:

Removing the fender liner revealed the fasteners underneath, this car was repainted at some point, possibly the shop that did it kept these down here to not lose track of where they came from. Good effort. Now only a handful are missing and are on the way from Amazon.

It took me a while to track down an annoying squeak: turns out it was coming from the hood latch.

I tried greasing it which helped for a small amount of time. There is some play in the mechanism you use to unlatch the hood but I don't immediately see how I could adjust it without adding washers. Perhaps the problem is how much the chassis flexes in general. I plan to stiffen the car a few different ways, including shock tower bracing that looks something like this.

This sort of bracing is OEM in some Miatas, but not this one. For now, I'm just going to make my hood latch not be metal by wrapping it in some electrical duct tape. It's not great, but I have bigger fish to fry.

Speaking of annoying noises, there was a rusty metal rattling coming from the exhaust, and I didn't need too much of an excuse to replace the exhaust of a vehicle. I opted for a higher-end Borla cat-back exhaust.

I struggled tremendously to get the original oxygen sensor out. I used 3/4 of the techniques in my arsenal. I started by hitting it with some penetrating fluid. Upgrading to a "cheater bar". I even tried dremelling out the old threads to create a slight relief channel. I stopped short of using MAP gas to heat the threads because I couldn't find my blow torch. Ultimately I just the wire on the old oxygen sensor and grabbed a fresh one from Autozone. So far the exhaust sounds great, it's a bit understated but I'll probably need to wait a couple hundred miles before I know what the real tone will be.

And that's some of the low-hanging fruit on the exterior of the car. Next up we're going to tackle this crusty old interior.

Updating the Interior

After addressing some of the higher priority mechanical items, it's time to improve the interior of this car. I'm going to start with the steering wheel, then I'll replace the seats, and finally I'll rebuild the shifter assembly.

Steering wheel

Presently, as I drive, the steering wheel deposits 25 year old leather crumbs into my hand. Let's try to replace it with something newer. I disconnected my battery so the airbag wouldn't deploy as I removed the old steering wheel.

Out came the old wheel, and in went an NRG quick release hub.

This hub allows the driver to detach the steering wheel prior to getting out of the car. The Miata is a small car, and once a race seat is fitted there will be limited thigh-room. Having the option to remove the wheel makes it easier to get in and out.

The final diameter of the steering wheel is much smaller which results in a sharper handling feel.

Seat

Next up is the seat: the current seat is torn in a few places. Additionally, it doesn't provide much support when cornering hard. Ultimately this car will have a 4 point harness and at that point changing the seat won't be optional.

I grabbed a pair of Sparco Sprint seats, with some Sparco mounting brackets. The Sparco Sprint seats fit in the NA Miata without having to modify the door cards or hammer in the transmission tunnel. Commonly, people will mount the seat directly to the car and give up the ability to adjust their seat. I want other people to be able to drive this car so I chose to mount the seat on the stock sliders.

I cleaned out and regreased the slider. After a mild amount of cutting and persuasion the seats were attached to the mounting brackets, and the mounting brackets were bolted to the sliders.

These seats are track oriented bucket seats. They provide strong side bostering to hold you in place when you're cornering hard. Donut media illustrated the difference between most stock seats and bucket seats in this video (jump to the end to see how bucket seats perform).

I'm not sure exactly what classes I'll be competing in long term, but thankfully these seats are FIA certified until 2028, which provides me with a large amount of flexibility.

Shifter assembly

Like the steering wheel, the shifter knob disentegrates everytime I glance at it. The knob presently rotates freely, and heat radiates from the leather boot which is torn.

I disassembled the entire shifter assembly, finding every layer to be in bad shape along the way. I also found Teflon tape under the universal shift knob.

In the shifter assembly there's nylon bushings which degrade over time, I grabbed a shifter rebuild kit and replaced all the nylon pieces.

This made the shfiter feel more precise and notchy. I also replaced all the rubber boots, and added fresh heat insulation. Because of the rebuild and the transmission service we performed earlier, the shifter now feels like it did 25 years ago.

With these 3 changes the interior is starting to be a very pleasant place to be.

Outstanding items

• Our windshield has a big chip in it.
• Oil change, and fuel filter replacement.
• Driver's side mirror wobbles a lot.
• To race, we need a roll bar in many classes, and a 4 point harness in some classes.
• There's a vibration at 85+mph. And there's a squeal during deceleration. I suspect the driveshaft.
• The stock suspension is very soft, I plan to install adjustable track oriented suspension and install harder polyurethane bushings in place of the 25 year old rubber bushings - the inherent softness of rubber has not been helped by age.
• The speakers sound very bad, and the head unit decides to beep a strange Sony theme song as a goodbye every time the car turns off.
• The windows move very slowly.
• The prior owner broke his key in the ignition, and as a fix decided to replace just the ignition assembly without changing the other locks in the car. So the car has 1 set of keys for the doors and a separate key for the ignition.
• The brakes fade very quickly, the pads and fluid may be at the end of their life. But it's common to just upgrade to Flyin' Miata's Big Brake Kit for a larger thermal capacity.
• Once all of these issues are sorted out, we'll embark on the journey of making more power.

Windshield Repair, Faster Windows, and More

This weekend we're going to knock out some non-glamorous maintainence items:

• Address the chip in the windshield
• Make the window go up and down faster
• Change the fuel filter and oil

Windshield chip

While I was driving on the highway a rock flew up and hit my windshield. I considered having Safelight repair it. So far I've figured out how to do everything on a car myself, so I was curious if I could do this as well. I compared a few windshield repair kits I went for one from Permatex.

The process involved using the syringes and the pedestal included in the kit to inject a resin into the damaged area. This damage is probably towards the extreme edge of what this kit can handle, so I don't expect perfection. I'd just like it to be less distracting and less likely to grow into a larger problem.

Overall this went pretty well. It's not flawless and you can spot it if you know what you're looking for. But it's very easy to register it as a smudge and ignore it. On a smaller chip I think the results would be closer to perfect.

Window speed

It takes almost 20 seconds for my passenger window to roll up and down, and it takes about 15 seconds for the drivers side. And so, into the door we go to see what's going on. If everything is intact, we'll just clean and lubricate the path the window takes. If anything is broken I expect it to be the window bushings. So before I got started I grabbed a set of replacements just in case.

My dad and I pealed back the layers of the door, and on the drivers side just cleaning and lubricating was all we had to do. Doing this brought down the window travel time to 5 seconds. On the passenger side when we pulled out the window we saw that a bushing was missing.

At the bottom of the door we saw the many pieces that used to make up the old bushing. I fished them out while my dad applied the new bushing. Like the driver's side we cleaned and greased the rails. Once the door was back together the window went down in 5 seconds like the driver's side. Success!

Getting into the door was a bit of a hassle, mainly due to the vapor barrier that's adhered to the door. Unfortunately, looking at our outstanding issues, this won't be the last time we have to get in here.

Other maintainence

I got a new Bosch fuel filter. A pretty straightforward procedure, but I'm not sure it's possible to do without getting gasoline on yourself.

When I went to change the engine oil, I found that the old oil filter was surprisingly difficult to remove on a car that's otherwise been very easy to work on. Always a bummer when you have to wait for a tool to show up to complete a job. But ultimately, with the new tool, the old oil was drained, and a round of fresh oil was poured. I used an oil filter with a nut on top just to give me another option for removal in the future.

Outstanding items

• Our windshield has a big chip in it.
• Oil change, and fuel filter replacement.
• Driver's side mirror wobbles a lot.
• To race, we need a roll bar in many classes, and a 4 point harness in some classes.
• There's a vibration at 85+mph. And there's a squeal during deceleration. I suspect the driveshaft.
• The stock suspension is very soft, I plan to install adjustable track oriented suspension and install harder polyurethane bushings in place of the 25 year old rubber bushings - the inherent softness of rubber has not been helped by age.
• The speakers sound very bad, and the head unit decides to beep a strange Sony theme song as a goodbye every time the car turns off.
• The windows move very slowly.
• The prior owner broke his key in the ignition, and as a fix decided to replace just the ignition assembly without changing the other locks in the car. So the car has 1 set of keys for the doors and a separate key for the ignition.
• The brakes fade very quickly, the pads and fluid may be at the end of their life. But it's common to just upgrade to Flyin' Miata's Big Brake Kit for a larger thermal capacity.
• Once all of these issues are sorted out, we'll embark on the journey of making more power.

Autocross 2023

Last season I raced the Miata, twice! I had a lot of fun and learned a great deal.

If Formula 1 racing is the top rung of motorsports, autocross is probably at the bottom. Autocross is generally hosted in a parking lot and the track is defined by cones. The format of the "race" is a time trial, one car is sent out at a time and the fastest time wins. You receive a time penalty for each cone you hit.

Autocross is one of the most accessible forms of racing. You can show up with just about any car, and compete against people in similar cars. The course setup and generally low overall speed minimize the chance of property damage.

This event is hosted by BMW CCA, and the classifications, as a result, are amusing: all non-BMW cars are in the same class. As a result, our Miata is in the same class as this open-wheel racecar:

Nor does the Miata qualify for any points, but about 50% of the cars there weren't BMWs, because racing cars is fun as hell.

I did two sessions to understand how our bone stock car stacks up. In Autocross the most competitive mods are generally great tires and suspension. The Miata does come with a reasonably competitive suspension from the factory. The suspension design: a double wishbone suspension is generally something you find on high-end sports cars.

So that's good, but our suspension is 27 years old, and if you look closely much of the rubber boot is frayed:

There are other pretty "tired" components like our bushings that are likely holding us back as well.

The tires we're running are also possibly the worst tires for this setting. They're all-weather hard-compound tires that are pretty practical for life as a daily car. But it is very impractical for racing cars around a parking lot (a very practical activity).

But nonetheless, Autocross is a very welcoming environment so I gave it a shot.

This event is particularly lax in the way that they don't require convertibles to install a roll cage. Tech-inspection is primarily an individual responsibility which is genuinely rare in motorsports. With the setup of this event, I also got to bring a buddy who could drive the Miata without reducing any of my seat time. For the first session, I brought my Dad who very quickly discovered that racing seats are uncomfortable. And the second time I brought my best friend Travis who took a handful of these photos.

Quickly after arriving you realize that not only will you be driving your car, but you'll also be a volunteer for running the event. Volunteers run out onto the course to fix downed cones and report when cars go off track. It's not always clear where the cone goes...

Even at low speeds the process of trying to improve your time lap after lap is quite fun. It rapidly becomes an exercise in memory, if you're thinking about where to go you're probably not doing as well as you could. Hesitation at the limit of traction probably means you're going to hit a cone or go off track. A few cones were destroyed by me:

Between the first and second sessions, I installed a bunch of telemetry in the car, to better understand how I'm doing, but also give me some time between sessions to try to remember the track better:

TODO VIDEO

After the event concludes race results are released: https://www.njbmwcca.org/event_info/results/autox_20231008_fin.htm. During our second, we completed 47th overall with 60 total contestants. And within our (relatively meaningless) class we were 30th out of 37th.

It's fascinating to see how all these cars stack up, and something I found particularly inspiring is that the value of our car is probably one of the lowest in the event overall, and the first session was won overall by a Miata.

It was also fascinating to see the incredible variety of cars at the event:

There were some dedicated, prepped, exotic track cars:

Some expensive cars:

Some unsuspecting stuff:

And some adorable stuff:

Overall I had a blast and I look forward to bringing more of my buddies to this eclectic weekend event, building the car, and building my skills up.

What's next for the car, is as expected: fresh tires, and fresh suspension. Before the next event, this is what we'll be taking care of.

Small Toolkit

• Built around this driver and this bag
• Bits
• Sockets
• Utility Knife
• Headlamp
• Tiny notebook
• Tiny tape measure*
• Tiny level, strong magnet, and stud-finder
• Super Snips
• Carpenter's Pencil

The toolkit that lives in my car.

considering exploring this laser tape measure.

Books

Books have had a large influence on my life. When I was young my dad made the value of reading clear to me. While reading was always something I did for fun, it was during college that reading started to feel transformative.

Unlike most, I organize the physical bookshelf in my house in the order I read the books, and here is the narrative I hold in my head as reflect upon what I've read and how it's influnced my life. The narrative is loosely held as at the time of choosing what to read, it's not really clear what the narrative is.

Before College

Before getting to college I was reading mostly for fun or because I had to from school. I enjoyed reading

• Harry Potter
• The Series of Unfortunate Events
• The Giver
• Percy Jackson trilogy
• 1984

College

Interestingly for Rutgers BS CS there was no required reading. And I was pretty hungry to learn about the world. I discovered a love for programming at an early age but I sorta built whatever the first idea that popped into my head was. I met one of my dad's friends who was tangentially involved in a high frequency trading firm. Until this point I believed that most programmers were just making Android and iOS apps. He recommended I read Flash Boys which began my first serious foray into reading non-fiction.

• Flash Boys
• Zero to One
• Lean Startup
• Crossing the Chasm
• Economics in One Lesson
• The Innovator's Dilemma
• Black Swan
• Rich Dad Poor Dad
• Hooked: How to Build Habit-Forming Products
• Fahrenheit 451
• Industrial Society and It's Future
• Elon Musk (Ashlee Vance)
• The Big Short

At the time I was an intern at SAP, a pretty large software company (comparable to Oracle in lots of ways) and I was working on a Robotics oriented startup. I was hungry to learn the processs of novel-value creation and spent a lot of time reflecting on what human progress actually means. As I generally gained confidence in myself I was actively trying to determine where I wanted my career to head. Did I want to try to be an entrepreneur? An academic? A politician? A highly paid software engineer? I started to form the desire to zoom out and understand the bigger picture more deeply, I think this desire was probably sparked after reading Black Swan. Towards year 3 & 4 of college it seemed like political tensions were also on the rise, and as far as I can tell, henceforth it's always felt that way. I wonder, has it just always been that way, and it's just my awareness that's changing. Anyways I also wanted to be good in the grand scheme of things and I was humble enough to know that it's not clear how to do that. I was open to utilitarian ideas of effective altruism, spritiual ideas of just being the best version of yourself, and seductive ideas of social change.

• 12 Rules for Life
• Ready Player One
• Sapiens: A Brief History of Humankind
• Principles: Life & Work
• Antifragile: Things that Gain from Disorder

After College

Towards the end of senior year I also had the good forture to experience a psychedelic trip with some of the people I respect the most in life. On graduating I was extended a full time offer from SAP, and soon after I would have a much more attractive offer from JP Morgan. On graduating, I lived in Jersey City with some of my closest friends, and the journey into self discovery continued. I read about traditional accounts of psychedlics, psychology, and neuroscience.

• Waking Up (Sam Harris)
• Brave New World
• Island
• Doors of perception
• Thinking Fast and Slow
• Outliers

While SAP was a big company, I worked on a small team that felt like it was solving experimental problems. Or maybe that's just what they had interns do. My insights during this time were largely about programming itself, and were of the straightforward kind. I was learning about React, APIs, databases. When I got to JP Morgan I experienced very little technological growth, but I learned a lot about working in social environments. At this time I started reading Ayn Rand and as someone who's read a lot of dystopian friction I've never read a more prophetic book.

• Atlas Shrugged.
• The Fountainhead
• Dune
• Dune Messiah
• Hitchiker's guide to the galaxy

It was during this time I transitioned to Gemini, an environment where I was surrounded by great peers, great mentors, and great technology. At this point I was pretty convinced that progress, and empowerement of the individual were very related ideas. Gemini had an environment which promoted a healthy exchange of ideas (which was rare, in my opinion). I wanted to learn more deeply about other tools that empower the individual (like crypto-currencies do), and I wanted to learn more deeply about the state. Gemini also re-ignited the idea that programming is a craft.

• Red Queen
• Last Call (on prohibition)
• Steve Jobs
• The anatomy of the state
• Come and take it (on 3d printed guns)
• The anarchist handbook
• Zen and art of Motorcycle Maintainance

After Gemini

I was captured by the idea that some stories are deeper than others. A book like Harry Potter has it's roots in the Cristian tradition. I had a desire to go to the source and read deeper content.

• Siddhartha
• Moby Dick
• 2001 A Space Odyssey
• Crime and Punishment
• Anna Karenina
• East of Eden
• Name Place Animal Thing
• Bhagavad Gita
• Plato
• Old Testiment
• Lolita

2025 onwards

It's hard and maybe not in my best interest to form a narritive about things that I'm actively reading, I sorta just want to read whatever interest me the most, but they'll continue to be listed here:

• Count of Monte Cristo

Reading Backlog

If someone recomends a book to me, it ends up here:

• The Gulag Archipelago
• Mountains Beyond Mountains (Carolyn)

• The Machiavellians: Malice
• A renegade history of the united states: Malice
• Dear Reader: Malice
• When Genius Failed: Steve, Swanson
• Society of mind: Lee
• GEB: Sid
• Win firends Influence People
• Origin of species
• Ben franklin
• War of art: Dahlia
• Ordinary Men: Peterson
• Paradise Lost: Peterson
• Panzram Journal of: Peterson
• Discovery of unconscious: Peterson
• Interpretation of dreams: Peterson
• Miracle morning: Steve
• Sleep: Raayan
• A mind for numbers: Alan
• Creature from Jekyll Island: Steve
• The day of the Jackal thriller: Dad
• The golden gate: Dad
• Arthur C Clarke moondust: Dad
• Asimov robot short stories: Dad
• Pg Wodehouse J&W: Dad
• Tao te Ching Raayan
• The mom test: Lee

Object Cache

I find myself in a tension between my desire to stay minimal and prepared. After I described my behavior my best friend compared it to a cache from CS, and that mental model stuck with me. This structure implies a nesting, I wouldn't be in my car without my backpack.

L1 Body

Always:

1. knife
2. phone
3. wallet
4. watch
5. modl bracelet

Sometimes:

• Flashlight
• Tiny Notebook
• Handgun

L2 Backpack

Always:

1. mbp
2. iPad (mini / pro)
3. charger + battery
4. 2-3 usb-c cables
5. airpods
6. flashlight
7. multitool
8. gum
9. hand-wipes
10. summer: packable rain layer / winter: packable down layer

Sometimes:

• handgun
• toiletry kit
• Yeti water bottle (small, medium) * (clear, insulated)
• water bladder

L3 Truck: 2017 F150 XLT SuperCrew

center console / interior:

• Jump start pack
• Multiple chargers + batteries plugged in by default
• Lighting & Usb C cables
• First aid and Trauma Kit
• Travel Toolkit
• Umbrella

under-seat:

• 8 retractable ratchet straps
• twine
• extension chord

in-doors:

• gloves
• tiny water bottles
• towels

truck bed (Organized via Molle panels + dry bags):

• Toiletry Kit & Change of clothes
• Recovery gear
• Hand Winch
• 2 gallons backup fuel
• 1+ gallon of water
• Tire inflator
• snow chains
• dog bag
  • Extra leash
  • Extra harness
  • Extra treats
  • Poop bags
  • Water bowl

Listed here are things that have been working well for some while. If something's missing I just don't have a solution that I think feels good.

Versatile

• Out and about town, or working at home, or around the house. No fancier socializing.
• Boots
  • Light, waterproof, and durable
• Socks
• Cargo Pants
  • Despite their name I've ripped quite a lot of Carhartt Ripstop workpants, so I'm trying these amazon basic ones. The fit is still a bit large on me (despite getting their smallest size), and so I am also experimenting with suspenders.

Heavy

• Boots
  • heavy but comfortable
  • I appreciate how quickly I can get in and out of them
  • so far home ownership has been a battle against water, and I appreciate how waterproof these are

• Overall Bib
  • These have lasted forever
  • I insert these kneepads into the washouts
  • Phone clipped to chest
• Gloves