Scaling Your Python CLI

How to maker your programs modular

As our programs grow we often need to go back to the drawing board

So you read some tutorials on building a CLI in python. You downloaded Click, created some groups and now have working CLI!

Then you start thinking of extra commands and realize it won’t scale. Adding everything into one file gets messy and the thought of multiple teams working on it sends shivers down your spine. In this article we’ll focus on two approaches to scale your CLI, making it more modular and easier to manage. Included is a Github repo with the initial project and various refactored ones.

This tutorial assumes you have an understanding of how Click works, but in case you don’t, you can read the following three articles to get up to speed:

• Click Docs
• Definitive Guide To Click
• Mastering Click

Our CLI

The test cli we will be building will contain three actions:

• create
• delete
• get

Each action can be executed on two targets:

• file
• directory

V1 Project Layout

To build our first cli version we followed a tutorial and put everything in one file. Not the best practice… but it worked! Our project looks like this.

├── main.py
├── setup.py

You can find the code in the V1 folder of the github repo. We’re using group decorators to structure our hierarchy as seen below.

Our general click hierarch

This works and we can compile our CLI, but adding extra functionality becomes more cluttered, so we try to refactor the code to be more modular.

V1.5 Trying To be Modular

We refactor our code and add a file.py and directory.py . We copy and paste our functions with decorators into them, but now run into a challenge, how do we structure the imports?

We try importing our top level CLI group into the submodules and adding commands that way, but we get empty sub commands. It’s time to reconsider our options.

V2 Modular Project Layout

To help create a more scalable approach we’re going to add python modules to our directory. Our project layout will mirror our categories (nouns), with a top level folder containing the cli entry point, two modules and a setup file.

├── main.py
├── setup.py
└── src
    ├── directories
    │   └── directory.py
    └── files
        └── file.py

Now we have a choice on how we’d like our users to interact with our cli, either structuring it so that commands go verb first or noun first, i.e cli create file or cli file create. In the next section we’ll discuss some pros and cons of each approach.

Structuring Your Commands

Depending on which CLIs you have worked with, the natural hierarchy for your cli might be cli create file or cli file create . The first one is called a verb-noun hierarchy with clis like Powershell. These structures are great when commands have similar actions that can be taken and mirrors english more closely.

The second type of noun-verb clis have more examples including the AWS CLI and linux bash commands. In the case of AWS, each service is the second part of each cli command such as aws ec2 describe instances and aws s3 ls.

The main benefit of this cli structure is it allows teams to contribute to a larger cli without changing the top level actions that can be taken. Teams can work more autonomously and users can choose which submodules to download. It also follows good object oriented principles more closely by not over relying on common methods in sub classes (groups) when they are not needed.

Other CLIs mix and match the noun-verb and verb-noun structure, like kubectl with commands like

kubectl config get-contexts
kubectl config view

And also commands like

kubectl get pods
kubectl get services

We will cover how to do both setups today in the next section and you’re encouraged to implement your preferred approach for your project.

Noun First — Cli File Create

We will start off with Noun first, because it makes our setup easier. Within our main.py under the Noun First folder, you will notice how we structure our project, we import each group from our sub directories and add them as a command.

Noun First Project

This was pretty easy! One thing to note is that I’m placing everything in src folder and then in my setup.py importing everything in src as a pymodule.

├── main.py
├── setup.py
└── src
├── directories
│ └── directory.py
└── files
└── file.py

When we run the cli we should see something like this.

cli file
Usage: cli file [OPTIONS] COMMAND [ARGS]...
Options:
  --help  Show this message and exit.
Commands:
  create
  delete
  get

Verb First — Cli Create File

Our second approach is to have our cli verb first, such as cli create file .

This approach is a little trickier because we can’t just add the create command to our cli top level group, we need to merge them. For this reason we can write a helper function of merge commands.

def merge_commands(group, command_collection:CommandCollection):
    """ Set the group's commands to those in the collection"""
    new_commands = {}
    command_sources = command_collection.sources
    for group in command_sources:
        for command_name,command in group.commands.items():
            new_commands[command_name] = command

    group.commands = new_commands
    return group

For each top level verb we would create a command collection and then pass in our group to override its empty commands.

Whenever we want to add a new module (noun) for a command, we can add it to our sources parameter in the CommandCollection init and our program will take care of the rest! The cli command for create would look something like this.

cli create
Usage: cli create [OPTIONS] COMMAND [ARGS]...
Options:
  --help  Show this message and exit.
Commands:
  directory  Create directory
  file       Create file

Note: You need to create the verb groups (create, get, delete) in the main.py and the submodules!

Conclusion

And there we have it. In the tutorial we learned how to scale our python Click cli by breaking down our groups into modules and supporting two types of cli structures. As you continue to scale you will run into more interesting challenges, and I’d be interested to hear about them along the way