Skip to content
This repository was archived by the owner on Sep 12, 2024. It is now read-only.

Conversation

Samyak2
Copy link
Contributor

@Samyak2 Samyak2 commented Apr 19, 2022

Uses Flask-SQLAlchemy's built-in pagination feature.

Added some helpers in `utils.pagination` to make it easier to return
pagination info as part of the response
@Samyak2 Samyak2 added ✨ enhancement New feature or request 🛠️ backend labels Apr 19, 2022
@Samyak2 Samyak2 added this to the v0.7.0 milestone Apr 19, 2022
@netlify
Copy link

netlify bot commented Apr 19, 2022

Deploy Preview for frontend-sb ready!

Name Link
🔨 Latest commit 912beff
🔍 Latest deploy log https://app.netlify.com/sites/frontend-sb/deploys/62710f3c7e826700098208bc
😎 Deploy Preview https://deploy-preview-930--frontend-sb.netlify.app/
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site settings.

@Samyak2 Samyak2 added the don't merge PR which doesn't need to be merged right now label Apr 19, 2022
Previously, `get_mapper_obj_by_dashboard_ids` would get the list of all
KPIs for the given dashboard_id, the query would use the entire list
(could be 500+ in some cases) embedded in the query itself. After this
query is done, pagination would be applied which doesn't provide much
benefits in terms of DB performance.

Refactored it to use DashboardKpiMapper in the query itself.
- the filter ensures that the entire list of KPIs is not retreived at
  any point
- pagination added in previous commit ensures performance

The `dashboards` field returned by the API endpoint was not used in the
frontend, so it was removed. This meant that a lot of unnecessary code
could be removed from the endpoint. Note: there is another end point to
get the list of dashboards, so this is not necessary.

The logic of associating dashboards with KPIs was refactored:
- Created function `kpi_dashboard_mapper_dict` in `dashboard_controller`
  that creates a dict that can be used to get the list of dashboards for
  given KPIs. This avoids calling multiple of `get_mapper_obj...`
  functions just for this purpose in the view.
- The function is type safe even when the return type depends on a
  parameter. Used `typing.overload` for this.

BREAKING CHANGE: removed `dashboards` fields from GET `/api/kpi`
endpoint.
@Samyak2 Samyak2 force-pushed the feature/pagination branch from 919a834 to f2f0c4f Compare April 19, 2022 08:47
Samyak2 added 5 commits April 19, 2022 14:21
according to frontend requirement
Optimized query when a dashboard_id is given (most of the time)
- Does not pull in the full list of KPIs anymore
- Uses a DB join instead

Added a helper for extracting pagination params from request.args.

Formatted file.
When the error is on the client, the status code should be 400
Also changed pagination_args to take the default value as parameters -
useful in debugging.
- formatted using black and isort
- fixed type errors
    - check the value returned by `get_json()` is None before using it
        - returns a message with 400 if it's not a JSON
- fixed flake8 errors
- fixed docstrings
- create a new logger for the alert view module instead of using the
  root logger (`current_app.logger`)
- reduced total errors and warnings from 60+ to 0
@Samyak2 Samyak2 force-pushed the feature/pagination branch from 6b01dc6 to 867ba87 Compare April 20, 2022 07:00
Samyak2 and others added 16 commits April 20, 2022 16:34
Refactored `get_alert_list` function:
- stabilized types. Used type overloading
  (https://docs.python.org/3/library/typing.html#typing.overload) to
  define the return type of the function based on argument values.
- For example, when `as_obj` is True, the return type is a list of Alert
  objects. When it's `False` it is a list of Dicts.
- It's a Pagination object when pagination is enabled
- Fixed docstring to be flake8 compatible and fixed other flake8 issues

Refactored `get_alert_info`:
- changed parameter name `id` to `alert_id` since `id` is a built-in
  function
- changed exception message to include the alert ID
- made docstring flake8 compatible and removed unnecessary arg
  description.

Added pagination support to `get_alert_list` function.
Added a new module `chaos_genius.utils.search` that has a helper
function that extracts the `query` arguments from URL parameters and
makes an SQL alchemy filter for the given column.

Used this in the list endpoint of KPI, kpi-dashboard, data source and
alert APIs.
- moved common filters to a list above the branches
- the search_filter is appended to it if present
- formatted with black and isort
- added docstrings for all functions
- validate JSON before using it
- ignore blind except errors
Used globals to store the default and mapping in one place - adding a
new sort require change to one place only.

Added `sort_by` arg in the list endpoint. Validation is performed on it
too. The sorting is done by `get_dashboard_list` in the controller
instead of directly having it in the view.

Added missing docstring to `get_dashboard_list`.
The endpoints are:
- `/api/connection/used-types`: list of data source types currently in
  use
- `/api/dashboard/names`: list of all dashboard names
- `/api/alert/used-status-types`: list of all alert status types
- `/api/alert/used-channel-types`: list of all alert channel types
  currently in use
KPI list (`/api/kpi`):
- supports `dashboard_id` and `datasource_type` params
- Example:
  `http://localhost:5000/api/kpi?dashboard_id=1&datasource_type=Postgres`

Data source list (`/api/connection`):
- supports `datasource_type`
- Example: `http://localhost:5000/api/connection?datasource_type=Druid`

Alert list (`/api/alert`):
- supports `channel` and `active`
- Example: `http://localhost:5000/api/alert?channel=email&active=false`
- To consider `active` to be True, use `true`, `1`, or `True`
- This is not named `status` because that has a different meaning in the
  backend/DB model
Uses getlist [1] to take all values of the parameter instead of just the
first one. The query is changed to use `in_` instead of `==`.

Type conversions are applied to each element in the list as appropriate.
The `type` parameter of `getlist` is not used since it ignores values
that cannot be parsed to that type - we want to show an explicit error
instead.

[1]: https://werkzeug.palletsprojects.com/en/2.1.x/datastructures/#werkzeug.datastructures.MultiDict.getlist
Made changes to return the filter params are empty.

Now multiple values for a filter param can be sent
spererated by '&' or the values can be ',' separated
or a combination of both.
@Samyak2
Copy link
Contributor Author

Samyak2 commented Apr 27, 2022

TODO:

  • some formatting and linting fixes after 79328ca
  • check if the other endpoint in KPI view needs the same changes as in 79328ca

manassolanki and others added 4 commits May 3, 2022 15:49
Inconsistencies were introduced in 79328ca
and added TODO for refactor in the future
The new logic to handle multiple filters given as comma separated values
and as multiple params added in 79328ca
was not added to the get-dashboard-list endpoint.

This commit adds it there too.
@Samyak2 Samyak2 marked this pull request as ready for review May 3, 2022 11:11
@gitguardian
Copy link

gitguardian bot commented May 3, 2022

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request
GitGuardian id Secret Commit Filename
2745890 Generic High Entropy Secret 912beff .env View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secret safely. Learn here the best practices.
  3. Revoke and rotate this secret.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!

@gitpod-io
Copy link

gitpod-io bot commented May 3, 2022

@Samyak2 Samyak2 removed the don't merge PR which doesn't need to be merged right now label May 3, 2022
@Samyak2 Samyak2 merged commit 1e10a59 into develop May 3, 2022
@Samyak2 Samyak2 deleted the feature/pagination branch May 3, 2022 11:21
@Samyak2 Samyak2 changed the title Pagination in API Pagination May 3, 2022
@Samyak2 Samyak2 mentioned this pull request May 5, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants