Development Practices for APU APPS

1. Programming Language

Back-end: Python 3.6/Python 3.7/Python 3.8/Python 3.9/Python 3.10

To use the higher versions of the Python it is necessary to ensure Zappa and Lambda both are supporting it.

Framework: Flask (*preferred), Chalice

Note 1: if you are working on a project with a team of developers you must have a requirement file in your repository.

Note 2: For database manipulation, you are highly recommended to use SQLAlchemy. SQLAlchemy is an open-source SQL toolkit and object-relational mapper for the Python programming language. It allows you to write Python code in your project to map from the database schema to the applications’ Python objects. No SQL is required to create, maintain and query the database. SQLAlchemy is an implementation of the object-relational mapping (ORM) concept.

2. IDE / Editor

Note: If you are using the same editor / IDE within the team you can push the editor / IDE configurations and setting to the repository so that all your team members have the same experience.

3. Coding Guidelines

Python:

You must include the requirements.txt with library’s particular version in the project. Reference: Requirements files Google Style Guide: https://google.github.io/styleguide/pyguide.html

4. Source Code Repository

The source code repository must be hosted on BitBucket

WHAT IS BITBUCKET?

The bucket must be owned by BitBucket user ctiteam https://bitbucket.org/ctiteam/ What to include in .gitignore?

# Specifies intentionally untracked files to ignore when using Git

venv/
.vendor/
__pycache__/
.*
*~
*.sw[mnpcod]
*.log
*.tmp
*.tmp.*
log.txt
*.sublime-project
*.sublime-workspace
.vscode/
npm-debug.log*
.idea/
.sass-cache/
.tmp/
.versions/
coverage/
dist/
node_modules/
tmp/
temp/
hooks/
platforms/
plugins/
plugins/android.json
plugins/ios.json
www/
$RECYCLE.BIN/
.DS_Store
Thumbs.db
UserInterfaceState.xcuserstate
*.md5
*.map

README.md

The bucket must contain a README.md file containing below sections:

Getting Started
- Prerequisites
- Installing
Running the tests
Deployment
Built With (framework, Main Dependencies)
License (required by public repositories)

5. Authentication / Authorization

Authentication

All authentications must be in a stateless manner. Do NOT use sessions. To use CAS for authentication use CAS file in the development tools project hosted here: https://bitbucket.org/ctiteam/dev-tools/ To understand how CAS authentication works, you may refer to CAS Sequence diagram

Authorization

To authorize the users you can create a group in active directory and rely on CAS response XML, which then contains memberOf.

6. Testing

For API testing refer to http://kb.sites.apiit.edu.my/knowledge-base/api-documentation/

7. Build / Deployment

What is pipeline?

Integrated CI/CD for Bitbucket Cloud that’s trivial to set up, automating your code from test to production. Easy setup and configuration There are no CI servers to set up, user management to configure, or repos to synchronize. Just enable Pipelines with a few simple clicks and you’re ready to go. A unified experience with Bitbucket Stop jumping between multiple applications. Manage your entire development workflow within Bitbucket, from code to deployment. Automate dev to deployment Sufficient coverage gives you the confidence to deploy. Reduce human error and keep the team lean working on critical tasks. Support for every platform Build and test with any language or platform including Java, JavaScript, PHP, Ruby, Python, .NET Code and more. Configuration as code Store and manage your build configurations in a single bitbucket-pipelines.yml file. Only 7 lines of code to get you started. Map the branch structure Pipelines can be aligned with the branch structure, making it easier to work with branching workflows like feature branching or git-flow.

What is Zappa and why using it?

Zappa makes it super easy to build and deploy serverless, event-driven Python applications (including, but not limited to, WSGI web apps) on AWS Lambda + API Gateway. Think of it as “serverless” web hosting for your Python apps. That means infinite scaling, zero downtime, zero maintenance – and at a fraction of the cost of your current deployments! If you’ve got a Python web app (including Django and Flask apps), it’s as easy as:

$ pip install zappa
$ zappa init
$ zappa deploy

and now you’re server-less! Wow!

What do you mean “serverless”?

Okay, so there still is a server – but it only has a 40-millisecond life cycle! Serverless in this case means “without any permanent infrastructure.” With a traditional HTTP server, the server is online 24/7, processing requests one by one as they come in. If the queue of incoming requests grows too large, some requests will time out. With Zappa, each request is given its own virtual HTTP “server” by Amazon API Gateway. AWS handles the horizontal scaling automatically, so no requests ever time out. Each request then calls your application from a memory cache in AWS Lambda and returns the response via Python’s WSGI interface. After your app returns, the “server” dies. Better still, with Zappa you only pay for the milliseconds of server time that you use, so it’s many orders of magnitude cheaper than VPS/PaaS hosts like Linode or Heroku – and in most cases, it’s completely free. Plus, there’s no need to worry about load balancing or keeping servers online ever again. It’s great for deploying serverless microservices with frameworks like Flask and Bottle, and for hosting larger web apps and CMSes with Django. Or, you can use any WSGI-compatible app you like! You probably don’t need to change your existing applications to use it, and you’re not locked into using it. Zappa also lets you build hybrid event-driven applications that can scale to trillions of events a year with no additional effort on your part! You also get free SSL certificates, global app deployment, API access management, automatic security policy generation, precompiled C-extensions, auto keep-warms, oversized Lambda packages, and many other exclusive features! And finally, Zappa is super easy to use. You can deploy your application with a single command out of the box! Awesome! To use Zappa in BitBucket pipeline you need to have a file named zappa_settings.json in the root of the repository.

The deployment is in AWS environemnt (most probably) as so you need to have the networking settings in the zappa file. That includes subnet ID and security group. You MUST request a new security group for each project. This is necessary as the other security groups attached to database or EC2 or other resources are relying on security group ID. For each new zappa deployment you MUST request a new AWS S3 bucket and define a rational project name.

Sample zappa_settings.json:

{
    "dev": {
        "app_function": "app.app",
        "aws_region": "ap-southeast-1",
        "project_name": "PROJECT_NAME",
        "runtime": "python3.9",
        "s3_bucket": "MY_PROJECT_NAME_S3",
        "aws_environment_variables": { 
            "MY_ENV_VAR_KEY": "MY_ENV_VAR_VALUE"
         },
        "vpc_config": {
            "SubnetIds": ["subnet-1234567890", "subnet-1234567890", "subnet-1234567890"],
            "SecurityGroupIds": ["sg-1234567890"]
        },
        "timeout_seconds": 300,
        "keep_warm": false,
        "manage_roles": true,
        "role_name": "basic-lambda",
        "role_arn": "arn:aws:iam::00000000:role/basic-lambda"
    },
    "production": {
        "app_function": "app.app",
        "aws_region": "ap-southeast-1",
        "project_name": "PROJECT_NAME",
        "runtime": "python3.9",
        "s3_bucket": "MY_PROJECT_NAME_S3",
        "aws_environment_variables": { 
           "MY_ENV_VAR_KEY": "MY_ENV_VAR_VALUE"
         },
        "vpc_config": {
            "SubnetIds": ["subnet-1234567890", "subnet-1234567890", "subnet-1234567890"],
            "SecurityGroupIds": ["sg-1234567890"]
        },
        "timeout_seconds": 600,
        "manage_roles": true,
        "role_name": "basic-lambda",
        "role_arn": "arn:aws:iam::00000000:role/basic-lambda"
    }
}

For more information on other zappa settings visit https://github.com/Miserlou/Zappa

Setting up pipelines

Must do steps in pipeline:

Build Images

What build image to be used for each env?

Dockerfile with AWS CLI used for deployment for various builders. Reference: AWS CLI BitBucket and CTI Docker Repository for AWS CLI To use pipeline in the BitBucket repository you must enable it in the settings. This action can only be done by Bucket Admin. To set-up the pipeline you need to have a file called bitbucket-pipelines.yml in the root of the repository. Sample bitbucket-pipelines.yml:

8. Monitoring and Maintenance

Save the log in CloudWatch. Code has to be maintained to ensure continuous availability and security.

9. User Feedback Collection

User feedback collection is via APSpace Feedback