From 24dd79b5664124f6fe2ba7ae4791cfe4c08967fb Mon Sep 17 00:00:00 2001 From: Akhil Mohan Date: Mon, 5 Feb 2024 16:31:03 +0530 Subject: [PATCH 1/5] feat: added guides for new backend development --- .env.migration.example | 1 + .gitignore | 2 +- backend/docs/guide/create-feature-x.md | 105 +++++++++++++++++++++++ backend/docs/guide/folder-structure.md | 81 +++++++++++++++++ backend/scripts/generate-schema-types.ts | 35 ++------ backend/src/db/knexfile.ts | 2 +- 6 files changed, 195 insertions(+), 31 deletions(-) create mode 100644 .env.migration.example create mode 100644 backend/docs/guide/create-feature-x.md create mode 100644 backend/docs/guide/folder-structure.md diff --git a/.env.migration.example b/.env.migration.example new file mode 100644 index 000000000..4d1c8f9ef --- /dev/null +++ b/.env.migration.example @@ -0,0 +1 @@ +DB_CONNECTION_URI= diff --git a/.gitignore b/.gitignore index f3c03e814..07322c82f 100644 --- a/.gitignore +++ b/.gitignore @@ -6,7 +6,7 @@ node_modules .env.gamma .env.prod .env.infisical - +.env.migration *~ *.swp *.swo diff --git a/backend/docs/guide/create-feature-x.md b/backend/docs/guide/create-feature-x.md new file mode 100644 index 000000000..e6994557f --- /dev/null +++ b/backend/docs/guide/create-feature-x.md @@ -0,0 +1,105 @@ +# Guide on creating a feature in Infisical backend + +Let's say you want to implement a new feature, call it feature-x for name sake. These are steps to follow. + +## Database model change + +If there is a database change, we must first address this to generate the database schemas for us to use. + +| Create a `.env.migration` for setting the db connection uri for migration scripts or you can just export the env DB_CONNECTION_URI + +1. if you have a new table, then go to `/src/db/schemas/models.ts` and update `TableName` enum to have the new table name. +2. Then create a new migration file by running `npm run migration:new`, type the name. Keep it something related to what your about to do. For now `feature-x` +3. Now go to `/src/db/migrations/_.ts` +4. Here update both the function `up` and `down` to create/alter the postgres fields on migration up and to revert it back on migration down. [Keeping it idempotent](https://github.com/graphile/migrate/blob/main/docs/idempotent-examples.md). + +### Generate TS schemas + +Typically you would need to know write TS types for knex type sense. But we have automated this process + +1. Start the server +2. Run `npm run migration:latest` to apply all the changes to db +3. Run `npm run generate:schema`. This will generate the type and schema using [zod](https://github.com/colinhacks/zod) in `/src/db/schemas` folder. +4. Update the barrel export in `schema/index` and apply the new tables names in `/src/@types/knex.d.ts`. This will allow knex js to have typesense. + +## Business Logic + +With the database changes generated. Now let's create the APIs for `feature-x`. + +1. Run `npm run generate:component` +2. Select 1 that is service component +3. Type service name in dashcase. Like `feature-x` + +This will create a folder inside `/src/services` with `feature-x` and 3 files + +1. `feature-x-dal`: The Database Access Layer function +2. `feature-x-service`: The service layer where all bussiness logic happens +3. `feature-x-type`: Types used by feature-x + +There are more layers like for reusable shared function u can setup a file called `feature-x-fns` + +You can use the custom infisical function `ormify` inside `src/lib/knex` to do simple db operations inside DAL. + +## Connecting the service layer with server layer + +All the server related logic happens inside `/src/server`. To connect the service layer inside server layer we use fastify plugins for dependency injection + +1. Add the service type inside `fastify.d.ts` file below `service` namespace of a FastifyServerInstance type +2. Now go to `/src/server/routes/index.ts`, instantiate the `feature-x` required dependencies like DAL layer and service layer and then pass it to `fastify.register("service,{...dependencies})` +3. With this the service layer will be accessibile inside all routes under fastify service instance. It can be accessed with `server.services..` + +## Writing the routes + +1. To create a route component run `npm generate:component` +2. Select option 3 by typing it out and then type the router name in dashcase. +3. Provide the version number + +This will generate a router file inside `src/server/routes/v/` + +1. Add your logic to connect with service layer accordingly +2. Then import the router component inside the version folder index.ts. Example, If the router component was inside v1, import the the function inside `v1/index.ts` +3. Finally register it under proper prefix to access it. + +The above contains the backend folder structure. All the contribution towards backend must follow the rules + +- **scripts**: Contains all the reusable scripts used in backend automation like running migration, generating SQL schemas +- **e2e-test**: The integration test for the APIs +- **src**: Source code of backend + +## Src + +- **@types**: The type definition of some libraries like fastify, knex +- **db**: Knexjs configuration required for database. Includes migration, seed files and sql type schemas +- **lib**: Stateless reusable functions used throught code base +- **queue**: Infisical queue system based on bullmq + +### Server + +- Anything related to fastify/service should be scoped inside here. +- It contains the routes, fastify plugins, server configurations +- Routes folder contains various version of routes seperated into v1,v2 + +### Services + +- Core bussiness logic for all operations +- Each service component follows co-location principle that is related things should be kept together +- Each service component contains +1. **dal**: The Database Access Layer function that contains all the db operations +2. **service**: The service layer containing all the bussiness logic +3. **type**: The type definition used inside the service component +4. **fns**: Optional component to share reusable functions from a service related to another +5. **queue**: Optional component to put queue specific logic for a component like `secret-queue.ts` + +## EE + +- Follows same pattern as above with an exception of licensn change from MIT -> Infisical Properitary License + +### Notes + +- All the services are interconnected at `/src/server/routes/index.ts`. We follow simple dependency injection principle +- All files should be in dashcases. +- Classes should not be used in codebase. Use simple functions to keep it simple +- All commited code must be linted properly by running `npm run lint:fix` and type checked using `npm run type:check` +- Try to avoid inter service shared logic as much as possible +- A controller inside a router component should try to keep it calling only one service layer. This rule could have exception when another service +like `audit-log` needs access to request object data. Then controller will call both the functions diff --git a/backend/docs/guide/folder-structure.md b/backend/docs/guide/folder-structure.md new file mode 100644 index 000000000..a86700106 --- /dev/null +++ b/backend/docs/guide/folder-structure.md @@ -0,0 +1,81 @@ +# Folder structure + +``` +. +├── scripts +├── e2e-test +└── src/ + ├── @types/ + │ ├── knex.d.ts + │ └── fastify.d.ts + ├── db/ + │ ├── migrations + │ ├── schemas + │ └── seed + ├── lib/ + │ ├── fn + │ ├── date + │ └── config + ├── queue + ├── server/ + │ ├── routes/ + │ │ ├── v1 + │ │ └── v2 + │ ├── plugins + │ └── config + ├── services/ + │ ├── auth + │ ├── org + │ └── project/ + │ ├── project-service.ts + │ ├── project-types.ts + │ └── project-dal.ts + └── ee/ + ├── routes + └── services +``` + +The above contains the backend folder structure. All the contribution towards backend must follow the rules + +- **scripts**: Contains all the reusable scripts used in backend automation like running migration, generating SQL schemas +- **e2e-test**: The integration test for the APIs +- **src**: Source code of backend + +## SRC + +- **@types**: The type definition of some libraries like fastify, knex +- **db**: Knexjs configuration required for database. Includes migration, seed files and sql type schemas +- **lib**: Stateless reusable functions used throught code base +- **queue**: Infisical queue system based on bullmq + +### Server + +- Anything related to fastify/service should be scoped inside here. +- It contains the routes, fastify plugins, server configurations +- Routes folder contains various version of routes seperated into v1,v2 + +### Services + +- Core bussiness logic for all operations +- Each service component follows co-location principle that is related things should be kept together +- Each service component contains + +1. **dal**: The Database Access Layer function that contains all the db operations +2. **service**: The service layer containing all the bussiness logic +3. **type**: The type definition used inside the service component +4. **fns**: Optional component to share reusable functions from a service related to another +5. **queue**: Optional component to put queue specific logic for a component like `secret-queue.ts` + +## EE + +- Follows same pattern as above with an exception of licensn change from MIT -> Infisical Properitary License + +### Notes + +- All the services are interconnected at `/src/server/routes/index.ts`. We follow simple dependency injection principle +- All files should be in dashcases. +- Classes should not be used in codebase. Use simple functions to keep it simple +- All commited code must be linted properly by running `npm run lint:fix` and type checked using `npm run type:check` +- Try to avoid inter service shared logic as much as possible +- A controller inside a router component should try to keep it calling only one service layer. This rule could have exception when another service + like `audit-log` needs access to request object data. Then controller will call both the functions diff --git a/backend/scripts/generate-schema-types.ts b/backend/scripts/generate-schema-types.ts index 5d51a5164..68330613c 100644 --- a/backend/scripts/generate-schema-types.ts +++ b/backend/scripts/generate-schema-types.ts @@ -3,13 +3,9 @@ import dotenv from "dotenv"; import path from "path"; import knex from "knex"; import { writeFileSync } from "fs"; -import promptSync from "prompt-sync"; - -const prompt = promptSync({ sigint: true }); dotenv.config({ - path: path.join(__dirname, "../.env"), - debug: true + path: path.join(__dirname, "../../.env.migration") }); const db = knex({ @@ -94,17 +90,7 @@ const main = async () => { .orderBy("table_name") ).filter((el) => !el.tableName.includes("_migrations")); - console.log("Select a table to generate schema"); - console.table(tables); - console.log("all: all tables"); - const selectedTables = prompt("Type table numbers comma seperated: "); - const tableNumbers = - selectedTables !== "all" ? selectedTables.split(",").map((el) => Number(el)) : []; - for (let i = 0; i < tables.length; i += 1) { - // skip if not desired table - if (selectedTables !== "all" && !tableNumbers.includes(i)) continue; - const { tableName } = tables[i]; const columns = await db(tableName).columnInfo(); const columnNames = Object.keys(columns); @@ -124,16 +110,16 @@ const main = async () => { if (colInfo.nullable) { ztype = ztype.concat(".nullable().optional()"); } - schema = schema.concat(`${!schema ? "\n" : ""} ${columnName}: ${ztype},\n`); + schema = schema.concat( + `${!schema ? "\n" : ""} ${columnName}: ${ztype}${colNum === columnNames.length - 1 ? "" : ","}\n` + ); } const dashcase = tableName.split("_").join("-"); const pascalCase = tableName .split("_") - .reduce( - (prev, curr) => prev + `${curr.at(0)?.toUpperCase()}${curr.slice(1).toLowerCase()}`, - "" - ); + .reduce((prev, curr) => prev + `${curr.at(0)?.toUpperCase()}${curr.slice(1).toLowerCase()}`, ""); + writeFileSync( path.join(__dirname, "../src/db/schemas", `${dashcase}.ts`), `// Code generated by automation script, DO NOT EDIT. @@ -152,15 +138,6 @@ export type T${pascalCase}Insert = Omit; export type T${pascalCase}Update = Partial>; ` ); - - // const file = readFileSync(path.join(__dirname, "../src/db/schemas/index.ts"), "utf8"); - // if (!file.includes(`export * from "./${dashcase};"`)) { - // appendFileSync( - // path.join(__dirname, "../src/db/schemas/index.ts"), - // `\nexport * from "./${dashcase}";`, - // "utf8" - // ); - // } } process.exit(0); diff --git a/backend/src/db/knexfile.ts b/backend/src/db/knexfile.ts index ec7458da6..19be10110 100644 --- a/backend/src/db/knexfile.ts +++ b/backend/src/db/knexfile.ts @@ -7,7 +7,7 @@ import path from "path"; // Update with your config settings. dotenv.config({ - path: path.join(__dirname, "../../.env"), + path: path.join(__dirname, "../../../.env.migration"), debug: true }); export default { From dc146d0883ffd14f1f051d827062b29f44858555 Mon Sep 17 00:00:00 2001 From: Akhil Mohan Date: Mon, 5 Feb 2024 22:28:40 +0530 Subject: [PATCH 2/5] feat: fixed spelling errors --- backend/docs/guide/create-feature-x.md | 16 ++++++++-------- backend/docs/guide/folder-structure.md | 12 ++++++------ 2 files changed, 14 insertions(+), 14 deletions(-) diff --git a/backend/docs/guide/create-feature-x.md b/backend/docs/guide/create-feature-x.md index e6994557f..91206ebc9 100644 --- a/backend/docs/guide/create-feature-x.md +++ b/backend/docs/guide/create-feature-x.md @@ -33,7 +33,7 @@ With the database changes generated. Now let's create the APIs for `feature-x`. This will create a folder inside `/src/services` with `feature-x` and 3 files 1. `feature-x-dal`: The Database Access Layer function -2. `feature-x-service`: The service layer where all bussiness logic happens +2. `feature-x-service`: The service layer where all business logic happens 3. `feature-x-type`: Types used by feature-x There are more layers like for reusable shared function u can setup a file called `feature-x-fns` @@ -46,7 +46,7 @@ All the server related logic happens inside `/src/server`. To connect the servic 1. Add the service type inside `fastify.d.ts` file below `service` namespace of a FastifyServerInstance type 2. Now go to `/src/server/routes/index.ts`, instantiate the `feature-x` required dependencies like DAL layer and service layer and then pass it to `fastify.register("service,{...dependencies})` -3. With this the service layer will be accessibile inside all routes under fastify service instance. It can be accessed with `server.services..` +3. With this the service layer will be accessible inside all routes under fastify service instance. It can be accessed with `server.services..` ## Writing the routes @@ -70,36 +70,36 @@ The above contains the backend folder structure. All the contribution towards ba - **@types**: The type definition of some libraries like fastify, knex - **db**: Knexjs configuration required for database. Includes migration, seed files and sql type schemas -- **lib**: Stateless reusable functions used throught code base +- **lib**: Stateless reusable functions used through code base - **queue**: Infisical queue system based on bullmq ### Server - Anything related to fastify/service should be scoped inside here. - It contains the routes, fastify plugins, server configurations -- Routes folder contains various version of routes seperated into v1,v2 +- Routes folder contains various version of routes separate into v1,v2 ### Services -- Core bussiness logic for all operations +- Core business logic for all operations - Each service component follows co-location principle that is related things should be kept together - Each service component contains 1. **dal**: The Database Access Layer function that contains all the db operations -2. **service**: The service layer containing all the bussiness logic +2. **service**: The service layer containing all the business logic 3. **type**: The type definition used inside the service component 4. **fns**: Optional component to share reusable functions from a service related to another 5. **queue**: Optional component to put queue specific logic for a component like `secret-queue.ts` ## EE -- Follows same pattern as above with an exception of licensn change from MIT -> Infisical Properitary License +- Follows same pattern as above with an exception of license change from MIT -> Infisical Proprietary License ### Notes - All the services are interconnected at `/src/server/routes/index.ts`. We follow simple dependency injection principle - All files should be in dashcases. - Classes should not be used in codebase. Use simple functions to keep it simple -- All commited code must be linted properly by running `npm run lint:fix` and type checked using `npm run type:check` +- All committed code must be linted properly by running `npm run lint:fix` and type checked using `npm run type:check` - Try to avoid inter service shared logic as much as possible - A controller inside a router component should try to keep it calling only one service layer. This rule could have exception when another service like `audit-log` needs access to request object data. Then controller will call both the functions diff --git a/backend/docs/guide/folder-structure.md b/backend/docs/guide/folder-structure.md index a86700106..6b2af0487 100644 --- a/backend/docs/guide/folder-structure.md +++ b/backend/docs/guide/folder-structure.md @@ -45,37 +45,37 @@ The above contains the backend folder structure. All the contribution towards ba - **@types**: The type definition of some libraries like fastify, knex - **db**: Knexjs configuration required for database. Includes migration, seed files and sql type schemas -- **lib**: Stateless reusable functions used throught code base +- **lib**: Stateless reusable functions used through code base - **queue**: Infisical queue system based on bullmq ### Server - Anything related to fastify/service should be scoped inside here. - It contains the routes, fastify plugins, server configurations -- Routes folder contains various version of routes seperated into v1,v2 +- Routes folder contains various version of routes separate into v1,v2 ### Services -- Core bussiness logic for all operations +- Core business logic for all operations - Each service component follows co-location principle that is related things should be kept together - Each service component contains 1. **dal**: The Database Access Layer function that contains all the db operations -2. **service**: The service layer containing all the bussiness logic +2. **service**: The service layer containing all the business logic 3. **type**: The type definition used inside the service component 4. **fns**: Optional component to share reusable functions from a service related to another 5. **queue**: Optional component to put queue specific logic for a component like `secret-queue.ts` ## EE -- Follows same pattern as above with an exception of licensn change from MIT -> Infisical Properitary License +- Follows same pattern as above with an exception of license change from MIT -> Infisical Proprietary License ### Notes - All the services are interconnected at `/src/server/routes/index.ts`. We follow simple dependency injection principle - All files should be in dashcases. - Classes should not be used in codebase. Use simple functions to keep it simple -- All commited code must be linted properly by running `npm run lint:fix` and type checked using `npm run type:check` +- All committed code must be linted properly by running `npm run lint:fix` and type checked using `npm run type:check` - Try to avoid inter service shared logic as much as possible - A controller inside a router component should try to keep it calling only one service layer. This rule could have exception when another service like `audit-log` needs access to request object data. Then controller will call both the functions From 734736273811af0830230b54fb2561b3d2315ac7 Mon Sep 17 00:00:00 2001 From: Maidul Islam Date: Mon, 12 Feb 2024 21:49:15 -0500 Subject: [PATCH 3/5] rephrase new feature development guide --- backend/docs/guide/create-feature-x.md | 105 ----------------- backend/docs/guide/folder-structure.md | 58 +++++----- backend/docs/guide/how-to-create-a-feature.md | 106 ++++++++++++++++++ 3 files changed, 134 insertions(+), 135 deletions(-) delete mode 100644 backend/docs/guide/create-feature-x.md create mode 100644 backend/docs/guide/how-to-create-a-feature.md diff --git a/backend/docs/guide/create-feature-x.md b/backend/docs/guide/create-feature-x.md deleted file mode 100644 index 91206ebc9..000000000 --- a/backend/docs/guide/create-feature-x.md +++ /dev/null @@ -1,105 +0,0 @@ -# Guide on creating a feature in Infisical backend - -Let's say you want to implement a new feature, call it feature-x for name sake. These are steps to follow. - -## Database model change - -If there is a database change, we must first address this to generate the database schemas for us to use. - -| Create a `.env.migration` for setting the db connection uri for migration scripts or you can just export the env DB_CONNECTION_URI - -1. if you have a new table, then go to `/src/db/schemas/models.ts` and update `TableName` enum to have the new table name. -2. Then create a new migration file by running `npm run migration:new`, type the name. Keep it something related to what your about to do. For now `feature-x` -3. Now go to `/src/db/migrations/_.ts` -4. Here update both the function `up` and `down` to create/alter the postgres fields on migration up and to revert it back on migration down. [Keeping it idempotent](https://github.com/graphile/migrate/blob/main/docs/idempotent-examples.md). - -### Generate TS schemas - -Typically you would need to know write TS types for knex type sense. But we have automated this process - -1. Start the server -2. Run `npm run migration:latest` to apply all the changes to db -3. Run `npm run generate:schema`. This will generate the type and schema using [zod](https://github.com/colinhacks/zod) in `/src/db/schemas` folder. -4. Update the barrel export in `schema/index` and apply the new tables names in `/src/@types/knex.d.ts`. This will allow knex js to have typesense. - -## Business Logic - -With the database changes generated. Now let's create the APIs for `feature-x`. - -1. Run `npm run generate:component` -2. Select 1 that is service component -3. Type service name in dashcase. Like `feature-x` - -This will create a folder inside `/src/services` with `feature-x` and 3 files - -1. `feature-x-dal`: The Database Access Layer function -2. `feature-x-service`: The service layer where all business logic happens -3. `feature-x-type`: Types used by feature-x - -There are more layers like for reusable shared function u can setup a file called `feature-x-fns` - -You can use the custom infisical function `ormify` inside `src/lib/knex` to do simple db operations inside DAL. - -## Connecting the service layer with server layer - -All the server related logic happens inside `/src/server`. To connect the service layer inside server layer we use fastify plugins for dependency injection - -1. Add the service type inside `fastify.d.ts` file below `service` namespace of a FastifyServerInstance type -2. Now go to `/src/server/routes/index.ts`, instantiate the `feature-x` required dependencies like DAL layer and service layer and then pass it to `fastify.register("service,{...dependencies})` -3. With this the service layer will be accessible inside all routes under fastify service instance. It can be accessed with `server.services..` - -## Writing the routes - -1. To create a route component run `npm generate:component` -2. Select option 3 by typing it out and then type the router name in dashcase. -3. Provide the version number - -This will generate a router file inside `src/server/routes/v/` - -1. Add your logic to connect with service layer accordingly -2. Then import the router component inside the version folder index.ts. Example, If the router component was inside v1, import the the function inside `v1/index.ts` -3. Finally register it under proper prefix to access it. - -The above contains the backend folder structure. All the contribution towards backend must follow the rules - -- **scripts**: Contains all the reusable scripts used in backend automation like running migration, generating SQL schemas -- **e2e-test**: The integration test for the APIs -- **src**: Source code of backend - -## Src - -- **@types**: The type definition of some libraries like fastify, knex -- **db**: Knexjs configuration required for database. Includes migration, seed files and sql type schemas -- **lib**: Stateless reusable functions used through code base -- **queue**: Infisical queue system based on bullmq - -### Server - -- Anything related to fastify/service should be scoped inside here. -- It contains the routes, fastify plugins, server configurations -- Routes folder contains various version of routes separate into v1,v2 - -### Services - -- Core business logic for all operations -- Each service component follows co-location principle that is related things should be kept together -- Each service component contains -1. **dal**: The Database Access Layer function that contains all the db operations -2. **service**: The service layer containing all the business logic -3. **type**: The type definition used inside the service component -4. **fns**: Optional component to share reusable functions from a service related to another -5. **queue**: Optional component to put queue specific logic for a component like `secret-queue.ts` - -## EE - -- Follows same pattern as above with an exception of license change from MIT -> Infisical Proprietary License - -### Notes - -- All the services are interconnected at `/src/server/routes/index.ts`. We follow simple dependency injection principle -- All files should be in dashcases. -- Classes should not be used in codebase. Use simple functions to keep it simple -- All committed code must be linted properly by running `npm run lint:fix` and type checked using `npm run type:check` -- Try to avoid inter service shared logic as much as possible -- A controller inside a router component should try to keep it calling only one service layer. This rule could have exception when another service -like `audit-log` needs access to request object data. Then controller will call both the functions diff --git a/backend/docs/guide/folder-structure.md b/backend/docs/guide/folder-structure.md index 6b2af0487..0b76c4e23 100644 --- a/backend/docs/guide/folder-structure.md +++ b/backend/docs/guide/folder-structure.md @@ -1,7 +1,6 @@ -# Folder structure +# Backend Folder Structure Guide ``` -. ├── scripts ├── e2e-test └── src/ @@ -35,47 +34,46 @@ └── services ``` -The above contains the backend folder structure. All the contribution towards backend must follow the rules -- **scripts**: Contains all the reusable scripts used in backend automation like running migration, generating SQL schemas -- **e2e-test**: The integration test for the APIs -- **src**: Source code of backend +The following outlines the backend folder structure. All contributions to the backend should adhere to these guidelines: + +- **scripts**: This folder contains all reusable scripts for backend automation, such as running migrations and generating SQL schemas. +- **e2e-test**: Here you'll find integration tests for the APIs. +- **src**: This is the main directory for the source code of the backend. ## SRC -- **@types**: The type definition of some libraries like fastify, knex -- **db**: Knexjs configuration required for database. Includes migration, seed files and sql type schemas -- **lib**: Stateless reusable functions used through code base -- **queue**: Infisical queue system based on bullmq +- **@types**: This directory holds the type definitions for certain libraries, such as Fastify and Knex. +- **db**: In this folder, you'll find the Knex.js configuration necessary for database operations, including migration, seed files, and SQL type schemas. +- **lib**: This directory is for stateless, reusable functions used throughout the codebase. +- **queue**: This folder contains the Infisical queue system, which is based on BullMQ. ### Server -- Anything related to fastify/service should be scoped inside here. -- It contains the routes, fastify plugins, server configurations -- Routes folder contains various version of routes separate into v1,v2 +- This section is dedicated to anything related to Fastify/service and should be contained within this scope. +- It includes routes, Fastify plugins, and server configurations. +- The routes folder is organized into various versions, separated into v1, v2, etc. ### Services -- Core business logic for all operations -- Each service component follows co-location principle that is related things should be kept together -- Each service component contains - -1. **dal**: The Database Access Layer function that contains all the db operations -2. **service**: The service layer containing all the business logic -3. **type**: The type definition used inside the service component -4. **fns**: Optional component to share reusable functions from a service related to another -5. **queue**: Optional component to put queue specific logic for a component like `secret-queue.ts` +- This area handles the core business logic for all operations. +- Each service component adheres to the co-location principle, meaning related components are grouped together. +- Within each service component, you will find: + 1. **dal**: The Database Access Layer, containing all database operations. + 2. **service**: This is the service layer where all the business logic resides. + 3. **type**: Type definitions used within the service component. + 4. **fns**: An optional component for sharing reusable functions related to the service. + 5. **queue**: An optional component for queue-specific logic, such as `secret-queue.ts`. ## EE -- Follows same pattern as above with an exception of license change from MIT -> Infisical Proprietary License +- This follows the same organizational pattern as above, but with a notable change from the MIT License to the Infisical Proprietary License. ### Notes -- All the services are interconnected at `/src/server/routes/index.ts`. We follow simple dependency injection principle -- All files should be in dashcases. -- Classes should not be used in codebase. Use simple functions to keep it simple -- All committed code must be linted properly by running `npm run lint:fix` and type checked using `npm run type:check` -- Try to avoid inter service shared logic as much as possible -- A controller inside a router component should try to keep it calling only one service layer. This rule could have exception when another service - like `audit-log` needs access to request object data. Then controller will call both the functions +- All services are interconnected at `/src/server/routes/index.ts`, where we employ a straightforward dependency injection principle. +- File naming should use dash-case. +- Instead of classes, the codebase relies on simple functions to maintain simplicity. +- All code committed must be thoroughly linted using `npm run lint:fix` and type-checked with `npm run type:check`. +- Efforts should be made to minimize shared logic between services. +- Controllers within a router component should generally invoke only one service layer. Exceptions may occur, such as when a service like `audit-log` requires access to request object data, necessitating calls to multiple functions. \ No newline at end of file diff --git a/backend/docs/guide/how-to-create-a-feature.md b/backend/docs/guide/how-to-create-a-feature.md new file mode 100644 index 000000000..06a1b3610 --- /dev/null +++ b/backend/docs/guide/how-to-create-a-feature.md @@ -0,0 +1,106 @@ +# Guide to Creating a Feature in Infisical's Backend + +Suppose you're interested in implementing a new feature, let's call it "feature-x." Here are the steps you should follow: + +## Database Model Change + +If your feature involves a change in the database, you need to first address this to generate the necessary database schemas. + +- Create a `.env.migration` file to set the database connection URI for migration scripts, or alternatively, export the `DB_CONNECTION_URI` environment variable. + +1. If you're adding a new table, update the `TableName` enum in `/src/db/schemas/models.ts` to include the new table name. +2. Create a new migration file by running `npm run migration:new` and give it a relevant name, such as `feature-x`. +3. Navigate to `/src/db/migrations/_.ts`. +4. Modify both the `up` and `down` functions to create or alter Postgres fields on migration up and to revert these changes on migration down, ensuring idempotency as outlined [here](https://github.com/graphile/migrate/blob/main/docs/idempotent-examples.md). + +### Generating TS Schemas + +While typically you would need to manually write TS types for Knex type-sense, we have automated this process: + +1. Start the server. +2. Run `npm run migration:latest` to apply all database changes. +3. Execute `npm run generate:schema` to automatically generate types and schemas using [zod](https://github.com/colinhacks/zod) in the `/src/db/schemas` folder. +4. Update the barrel export in `schema/index` and include the new tables in `/src/@types/knex.d.ts` to enable type-sensing in Knex.js. + +## Business Logic + +Once the database changes are in place, it's time to create the APIs for `feature-x`: + +1. Execute `npm run generate:component`. +2. Choose option 1 for the service component. +3. Name the service in dash-case, like `feature-x`. + +This will create a `feature-x` folder in `/src/services` containing three files: + +1. `feature-x-dal`: The Database Access Layer functions. +2. `feature-x-service`: The service layer where all the business logic is handled. +3. `feature-x-type`: The types used by `feature-x`. + +For reusable shared functions, set up a file named `feature-x-fns`. + +Use the custom Infisical function `ormify` in `src/lib/knex` for simple database operations within the DAL. + +## Connecting the Service Layer to the Server Layer + +Server-related logic is handled in `/src/server`. To connect the service layer to the server layer, we use Fastify plugins for dependency injection: + +1. Add the service type in the `fastify.d.ts` file under the `service` namespace of a FastifyServerInstance type. +2. In `/src/server/routes/index.ts`, instantiate the required dependencies for `feature-x`, such as the DAL and service layers, and then pass them to `fastify.register("service,{...dependencies})`. +3. This makes the service layer accessible within all routes under the Fastify service instance, accessed via `server.services..`. + +## Writing the Routes + +1. To create a route component, run `npm generate:component`. +2. Select option 3, type the router name in dash-case, and provide the version number. + +This will generate a router file in `src/server/routes/v/`: + +1. Implement your logic to connect with the service layer as needed. +2. Import the router component in the version folder's index.ts. For instance, if it's in v1, import it in `v1/index.ts`. +3. Finally, register it under the appropriate prefix for access. + +## Backend Folder Structure + +Contributions to the backend must adhere to the following structure: + +- **scripts**: Contains reusable scripts for backend automation, like running migrations and generating SQL schemas. +- **e2e-test**: Integration tests for the APIs. +- **src**: The source code of the backend. + +### Src + +- **@types**: Type definitions for libraries like Fastify and Knex. +- **db**: Knex.js configuration for the database, including migration, seed files, and SQL type schemas. +- **lib**: Stateless, reusable functions used across the codebase. +- **queue**: Infisical's queue system based on BullMQ. + +### Server + +- Scope anything related to Fastify/service here. +- Includes routes, Fastify plugins, and server configurations. +- The routes folder contains various versions of routes separated into v1, v2, etc. + +### Services + +- Handles the core business logic for all operations. +- Follows the co-location principle: related components should be kept together. +- Each service component typically contains: + + 1. **dal**: Database Access Layer functions for database operations + 2. **service**: The service layer containing business logic. + 3. **type**: Type definitions used within the service component. + 4. **fns**: An optional component for sharing reusable functions related to the service. + 5. **queue**: An optional component for queue-specific logic, like `secret-queue.ts`. + +## EE + +- Follows the same pattern as above, with the exception of a license change from MIT to Infisical Proprietary License. + +### Notes + +- All services are interconnected at `/src/server/routes/index.ts`, following the principle of simple dependency injection. +- Files should be named in dash-case. +- Avoid using classes in the codebase; opt for simple functions instead. +- All committed code must be properly linted using `npm run lint:fix` and type-checked with `npm run type:check`. +- Minimize shared logic between services as much as possible. +- Controllers within a router component should ideally call only one service layer, with exceptions for services like `audit-log` that require access to request object data. \ No newline at end of file From ad70c783e8f141e2b5af9a08c109130ea59987fc Mon Sep 17 00:00:00 2001 From: Maidul Islam Date: Mon, 12 Feb 2024 22:33:55 -0500 Subject: [PATCH 4/5] add backend guide to contributor --- .../platform/backend/folder-structure.mdx | 82 +++++++++++++++++++ .../backend/how-to-create-a-feature.mdx | 56 +++++++++++++ docs/mint.json | 4 +- 3 files changed, 141 insertions(+), 1 deletion(-) create mode 100644 docs/contributing/platform/backend/folder-structure.mdx create mode 100644 docs/contributing/platform/backend/how-to-create-a-feature.mdx diff --git a/docs/contributing/platform/backend/folder-structure.mdx b/docs/contributing/platform/backend/folder-structure.mdx new file mode 100644 index 000000000..abfe0f69d --- /dev/null +++ b/docs/contributing/platform/backend/folder-structure.mdx @@ -0,0 +1,82 @@ +--- +title: 'Backend folder structure' +--- + +``` +├── scripts +├── e2e-test +└── src/ + ├── @types/ + │ ├── knex.d.ts + │ └── fastify.d.ts + ├── db/ + │ ├── migrations + │ ├── schemas + │ └── seed + ├── lib/ + │ ├── fn + │ ├── date + │ └── config + ├── queue + ├── server/ + │ ├── routes/ + │ │ ├── v1 + │ │ └── v2 + │ ├── plugins + │ └── config + ├── services/ + │ ├── auth + │ ├── org + │ └── project/ + │ ├── project-service.ts + │ ├── project-types.ts + │ └── project-dal.ts + └── ee/ + ├── routes + └── services +``` + +### `backend/scripts` +Contains reusable scripts for backend automation, like running migrations and generating SQL schemas. + +### `backend/e2e-test` +Integration tests for the APIs. + +### `backend/src` +The source code of the backend. + +- `@types`: Type definitions for libraries like Fastify and Knex. +- `db`: Knex.js configuration for the database, including migration, seed files, and SQL type schemas. +- `lib`: Stateless, reusable functions used across the codebase. +- `queue`: Infisical's queue system based on BullMQ. + +### `src/server` + +- Scope anything related to Fastify/service here. +- Includes routes, Fastify plugins, and server configurations. +- The routes folder contains various versions of routes separated into v1, v2, etc. + +### `src/services` + +- Handles the core business logic for all operations. +- Follows the co-location principle: related components should be kept together. +- Each service component typically contains: + + 1. **dal**: Database Access Layer functions for database operations + 2. **service**: The service layer containing business logic. + 3. **type**: Type definitions used within the service component. + 4. **fns**: An optional component for sharing reusable functions related to the service. + 5. **queue**: An optional component for queue-specific logic, like `secret-queue.ts`. + +### `src/ee` + +Follows the same pattern as above, with the exception of a license change from MIT to Infisical Proprietary License. + +### Guidelines and Best Practices + +- All services are interconnected at `/src/server/routes/index.ts`, following the principle of simple dependency injection. +- Files should be named in dash-case. +- Avoid using classes in the codebase; opt for simple functions instead. +- All committed code must be properly linted using `npm run lint:fix` and type-checked with `npm run type:check`. +- Minimize shared logic between services as much as possible. +- Controllers within a router component should ideally call only one service layer, with exceptions for services like `audit-log` that require access to request object data. \ No newline at end of file diff --git a/docs/contributing/platform/backend/how-to-create-a-feature.mdx b/docs/contributing/platform/backend/how-to-create-a-feature.mdx new file mode 100644 index 000000000..207cf7fa8 --- /dev/null +++ b/docs/contributing/platform/backend/how-to-create-a-feature.mdx @@ -0,0 +1,56 @@ +--- +title: "Backend development guide" +--- + +Suppose you're interested in implementing a new feature, let's call it "feature-x." Here are the steps you should follow: + +## Database schema migration +In order to run [schema migrations](https://en.wikipedia.org/wiki/Schema_migration#:~:text=A%20schema%20migration%20is%20performed,some%20newer%20or%20older%20version) you need to expose your database connection string. Create a `.env.migration` file to set the database connection URI for migration scripts, or alternatively, export the `DB_CONNECTION_URI` environment variable. + +## Creating new database model +If your feature involves a change in the database, you need to first address this by generating the necessary database schemas. + +1. If you're adding a new table, update the `TableName` enum in `/src/db/schemas/models.ts` to include the new table name. +2. Create a new migration file by running `npm run migration:new` and give it a relevant name, such as `feature-x`. +3. Navigate to `/src/db/migrations/_.ts`. +4. Modify both the `up` and `down` functions to create or alter Postgres fields on migration up and to revert these changes on migration down, ensuring idempotency as outlined [here](https://github.com/graphile/migrate/blob/main/docs/idempotent-examples.md). + +### Generating TS Schemas + +While typically you would need to manually write TS types for Knex type-sense, we have automated this process: + +1. Start the server. +2. Run `npm run migration:latest` to apply all database changes. +3. Execute `npm run generate:schema` to automatically generate types and schemas using [zod](https://github.com/colinhacks/zod) in the `/src/db/schemas` folder. +4. Update the barrel export in `schema/index` and include the new tables in `/src/@types/knex.d.ts` to enable type-sensing in Knex.js. + +## Business Logic + +Once the database changes are in place, it's time to create the APIs for `feature-x`: + +1. Execute `npm run generate:component`. +2. Choose option 1 for the service component. +3. Name the service in dash-case, like `feature-x`. This will create a `feature-x` folder in `/src/services` containing three files. + 1. `feature-x-dal`: The Database Access Layer functions. + 2. `feature-x-service`: The service layer where all the business logic is handled. + 3. `feature-x-type`: The types used by `feature-x`. + +For reusable shared functions, set up a file named `feature-x-fns`. + +Use the custom Infisical function `ormify` in `src/lib/knex` for simple database operations within the DAL. + +## Connecting the Service Layer to the Server Layer + +Server-related logic is handled in `/src/server`. To connect the service layer to the server layer, we use Fastify plugins for dependency injection: + +1. Add the service type in the `fastify.d.ts` file under the `service` namespace of a FastifyServerInstance type. +2. In `/src/server/routes/index.ts`, instantiate the required dependencies for `feature-x`, such as the DAL and service layers, and then pass them to `fastify.register("service,{...dependencies})`. +3. This makes the service layer accessible within all routes under the Fastify service instance, accessed via `server.services..`. + +## Writing API Routes + +1. To create a route component, run `npm generate:component`. +2. Select option 3, type the router name in dash-case, and provide the version number. This will generate a router file in `src/server/routes/v/` + 1. Implement your logic to connect with the service layer as needed. + 2. Import the router component in the version folder's index.ts. For instance, if it's in v1, import it in `v1/index.ts`. + 3. Finally, register it under the appropriate prefix for access. \ No newline at end of file diff --git a/docs/mint.json b/docs/mint.json index 1f8dc618b..3fa60bdac 100644 --- a/docs/mint.json +++ b/docs/mint.json @@ -462,7 +462,9 @@ { "group": "Contributing to platform", "pages": [ - "contributing/platform/developing" + "contributing/platform/developing", + "contributing/platform/backend/how-to-create-a-feature", + "contributing/platform/backend/folder-structure" ] }, { From 3c4d9fd4a9dff2247f2646c0c6847af87841215f Mon Sep 17 00:00:00 2001 From: Maidul Islam Date: Mon, 12 Feb 2024 22:34:36 -0500 Subject: [PATCH 5/5] delete docs in backend --- backend/docs/guide/folder-structure.md | 79 ------------- backend/docs/guide/how-to-create-a-feature.md | 106 ------------------ 2 files changed, 185 deletions(-) delete mode 100644 backend/docs/guide/folder-structure.md delete mode 100644 backend/docs/guide/how-to-create-a-feature.md diff --git a/backend/docs/guide/folder-structure.md b/backend/docs/guide/folder-structure.md deleted file mode 100644 index 0b76c4e23..000000000 --- a/backend/docs/guide/folder-structure.md +++ /dev/null @@ -1,79 +0,0 @@ -# Backend Folder Structure Guide - -``` -├── scripts -├── e2e-test -└── src/ - ├── @types/ - │ ├── knex.d.ts - │ └── fastify.d.ts - ├── db/ - │ ├── migrations - │ ├── schemas - │ └── seed - ├── lib/ - │ ├── fn - │ ├── date - │ └── config - ├── queue - ├── server/ - │ ├── routes/ - │ │ ├── v1 - │ │ └── v2 - │ ├── plugins - │ └── config - ├── services/ - │ ├── auth - │ ├── org - │ └── project/ - │ ├── project-service.ts - │ ├── project-types.ts - │ └── project-dal.ts - └── ee/ - ├── routes - └── services -``` - - -The following outlines the backend folder structure. All contributions to the backend should adhere to these guidelines: - -- **scripts**: This folder contains all reusable scripts for backend automation, such as running migrations and generating SQL schemas. -- **e2e-test**: Here you'll find integration tests for the APIs. -- **src**: This is the main directory for the source code of the backend. - -## SRC - -- **@types**: This directory holds the type definitions for certain libraries, such as Fastify and Knex. -- **db**: In this folder, you'll find the Knex.js configuration necessary for database operations, including migration, seed files, and SQL type schemas. -- **lib**: This directory is for stateless, reusable functions used throughout the codebase. -- **queue**: This folder contains the Infisical queue system, which is based on BullMQ. - -### Server - -- This section is dedicated to anything related to Fastify/service and should be contained within this scope. -- It includes routes, Fastify plugins, and server configurations. -- The routes folder is organized into various versions, separated into v1, v2, etc. - -### Services - -- This area handles the core business logic for all operations. -- Each service component adheres to the co-location principle, meaning related components are grouped together. -- Within each service component, you will find: - 1. **dal**: The Database Access Layer, containing all database operations. - 2. **service**: This is the service layer where all the business logic resides. - 3. **type**: Type definitions used within the service component. - 4. **fns**: An optional component for sharing reusable functions related to the service. - 5. **queue**: An optional component for queue-specific logic, such as `secret-queue.ts`. - -## EE - -- This follows the same organizational pattern as above, but with a notable change from the MIT License to the Infisical Proprietary License. - -### Notes - -- All services are interconnected at `/src/server/routes/index.ts`, where we employ a straightforward dependency injection principle. -- File naming should use dash-case. -- Instead of classes, the codebase relies on simple functions to maintain simplicity. -- All code committed must be thoroughly linted using `npm run lint:fix` and type-checked with `npm run type:check`. -- Efforts should be made to minimize shared logic between services. -- Controllers within a router component should generally invoke only one service layer. Exceptions may occur, such as when a service like `audit-log` requires access to request object data, necessitating calls to multiple functions. \ No newline at end of file diff --git a/backend/docs/guide/how-to-create-a-feature.md b/backend/docs/guide/how-to-create-a-feature.md deleted file mode 100644 index 06a1b3610..000000000 --- a/backend/docs/guide/how-to-create-a-feature.md +++ /dev/null @@ -1,106 +0,0 @@ -# Guide to Creating a Feature in Infisical's Backend - -Suppose you're interested in implementing a new feature, let's call it "feature-x." Here are the steps you should follow: - -## Database Model Change - -If your feature involves a change in the database, you need to first address this to generate the necessary database schemas. - -- Create a `.env.migration` file to set the database connection URI for migration scripts, or alternatively, export the `DB_CONNECTION_URI` environment variable. - -1. If you're adding a new table, update the `TableName` enum in `/src/db/schemas/models.ts` to include the new table name. -2. Create a new migration file by running `npm run migration:new` and give it a relevant name, such as `feature-x`. -3. Navigate to `/src/db/migrations/_.ts`. -4. Modify both the `up` and `down` functions to create or alter Postgres fields on migration up and to revert these changes on migration down, ensuring idempotency as outlined [here](https://github.com/graphile/migrate/blob/main/docs/idempotent-examples.md). - -### Generating TS Schemas - -While typically you would need to manually write TS types for Knex type-sense, we have automated this process: - -1. Start the server. -2. Run `npm run migration:latest` to apply all database changes. -3. Execute `npm run generate:schema` to automatically generate types and schemas using [zod](https://github.com/colinhacks/zod) in the `/src/db/schemas` folder. -4. Update the barrel export in `schema/index` and include the new tables in `/src/@types/knex.d.ts` to enable type-sensing in Knex.js. - -## Business Logic - -Once the database changes are in place, it's time to create the APIs for `feature-x`: - -1. Execute `npm run generate:component`. -2. Choose option 1 for the service component. -3. Name the service in dash-case, like `feature-x`. - -This will create a `feature-x` folder in `/src/services` containing three files: - -1. `feature-x-dal`: The Database Access Layer functions. -2. `feature-x-service`: The service layer where all the business logic is handled. -3. `feature-x-type`: The types used by `feature-x`. - -For reusable shared functions, set up a file named `feature-x-fns`. - -Use the custom Infisical function `ormify` in `src/lib/knex` for simple database operations within the DAL. - -## Connecting the Service Layer to the Server Layer - -Server-related logic is handled in `/src/server`. To connect the service layer to the server layer, we use Fastify plugins for dependency injection: - -1. Add the service type in the `fastify.d.ts` file under the `service` namespace of a FastifyServerInstance type. -2. In `/src/server/routes/index.ts`, instantiate the required dependencies for `feature-x`, such as the DAL and service layers, and then pass them to `fastify.register("service,{...dependencies})`. -3. This makes the service layer accessible within all routes under the Fastify service instance, accessed via `server.services..`. - -## Writing the Routes - -1. To create a route component, run `npm generate:component`. -2. Select option 3, type the router name in dash-case, and provide the version number. - -This will generate a router file in `src/server/routes/v/`: - -1. Implement your logic to connect with the service layer as needed. -2. Import the router component in the version folder's index.ts. For instance, if it's in v1, import it in `v1/index.ts`. -3. Finally, register it under the appropriate prefix for access. - -## Backend Folder Structure - -Contributions to the backend must adhere to the following structure: - -- **scripts**: Contains reusable scripts for backend automation, like running migrations and generating SQL schemas. -- **e2e-test**: Integration tests for the APIs. -- **src**: The source code of the backend. - -### Src - -- **@types**: Type definitions for libraries like Fastify and Knex. -- **db**: Knex.js configuration for the database, including migration, seed files, and SQL type schemas. -- **lib**: Stateless, reusable functions used across the codebase. -- **queue**: Infisical's queue system based on BullMQ. - -### Server - -- Scope anything related to Fastify/service here. -- Includes routes, Fastify plugins, and server configurations. -- The routes folder contains various versions of routes separated into v1, v2, etc. - -### Services - -- Handles the core business logic for all operations. -- Follows the co-location principle: related components should be kept together. -- Each service component typically contains: - - 1. **dal**: Database Access Layer functions for database operations - 2. **service**: The service layer containing business logic. - 3. **type**: Type definitions used within the service component. - 4. **fns**: An optional component for sharing reusable functions related to the service. - 5. **queue**: An optional component for queue-specific logic, like `secret-queue.ts`. - -## EE - -- Follows the same pattern as above, with the exception of a license change from MIT to Infisical Proprietary License. - -### Notes - -- All services are interconnected at `/src/server/routes/index.ts`, following the principle of simple dependency injection. -- Files should be named in dash-case. -- Avoid using classes in the codebase; opt for simple functions instead. -- All committed code must be properly linted using `npm run lint:fix` and type-checked with `npm run type:check`. -- Minimize shared logic between services as much as possible. -- Controllers within a router component should ideally call only one service layer, with exceptions for services like `audit-log` that require access to request object data. \ No newline at end of file