Extractions, calculations and optimizations + some documentation #6
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -23,7 +23,6 @@ repos: | |
| hooks: | ||
| - id: ruff | ||
| - id: ruff-format | ||
|
|
||
| - repo: local | ||
| hooks: | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,41 @@ | ||
| # programapi | ||
|
|
||
| This project downloads, processes, saves, and serves the static JSON files containing details of accepted speakers and submissions via an API. | ||
|
|
||
| Used by the EuroPython 2024 website and the Discord bot. | ||
|
|
||
| **What this project does step-by-step:** | ||
|
|
||
| 1. Downloads the Pretalx speaker and submission data, and saves it as JSON files. | ||
| 2. Transforms the JSON files into a format that is easier to work with and OK to serve publicly. This includes removing unnecessary/private fields, and adding new fields. | ||
| 3. Serves the JSON files via an API. | ||
|
|
||
| ## Installation | ||
|
|
||
| 1. Clone the repository. | ||
| 2. Install the dependency management tool: ``make deps/pre`` | ||
|
There was a problem hiding this comment. Maybe an idea for the future. What about we just install
There was a problem hiding this comment. I thought about it and didn't want to do it when there was bunch of other stuff to implement first. But I support it, I think we can do it after this PR. About the private/unused fields:
|
||
| 3. Install the dependencies: ``make deps/install`` | ||
| 4. Set up ``pre-commit``: ``make pre-commit`` | ||
|
|
||
| ## Configuration | ||
|
|
||
| You can change the event in the [``config.py``](src/config.py) file. It is set to ``europython-2024`` right now. | ||
|
|
||
| ## Usage | ||
|
|
||
| - Run the whole process: ``make all`` | ||
| - Run only the download process: ``make download`` | ||
| - Run only the transformation process: ``make transform`` | ||
|
|
||
| **Note:** Don't forget to set ``PRETALX_TOKEN`` in your ``.env`` file at the root of the project. And please don't make too many requests to the Pretalx API, it might get angry 🤪 | ||
|
|
||
| ## API | ||
|
|
||
| The API is served at ``https://programapi24.europython.eu/2024``. It has two endpoints (for now): | ||
|
|
||
| - ``/speakers.json``: Returns the list of confirmed speakers. | ||
| - ``/sessions.json``: Returns the list of confirmed sessions. | ||
|
|
||
| ## Schema | ||
|
|
||
| See [this page](data/examples/README.md) for the explanations of the fields in the returned JSON files. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| # JSON files except the ones in examples/ | ||
| *.json | ||
| !examples/** |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,139 @@ | ||
| # Explaining the output data | ||
|
|
||
| **Note:** Some of the fields may be `null` or empty (`""`). | ||
|
|
||
| ## `sessions.json` | ||
|
|
||
| <details> | ||
| <summary>Example session data JSON</summary> | ||
|
|
||
| ```json | ||
| { | ||
| "A1B2C3": { | ||
| "code": "A1B2C3", | ||
| "title": "Example talk", | ||
| "speakers": [ | ||
| "B4D5E6", | ||
| ... | ||
| ], | ||
| "session_type": "Talk", | ||
| "slug": "example-talk", | ||
| "track": "Some Track", | ||
| "state": "confirmed", | ||
| "abstract": "This is an example talk. It is a great talk.", | ||
| "tweet": "This is an example talk.", | ||
| "duration": "60", | ||
| "level": "intermediate", | ||
| "delivery": "in-person", | ||
| "resources": [ | ||
| { | ||
| "resource": "https://example.com/notebook.ipynb", | ||
| "description": "Notebook used in the talk" | ||
| }, | ||
| { | ||
| "resource": "https://www.youtube.com/watch?v=dQw4w9WgXcQ", | ||
| "description": "Video of the robot in action" | ||
| } | ||
| ... | ||
| ], | ||
| "room": "South Hall 2A", | ||
| "start": "2024-07-10T14:00:00+02:00", | ||
| "end": "2024-07-10T15:00:00+02:00", | ||
| "website_url": "https://ep2024.europython.eu/session/example-talk/", | ||
| "sessions_in_parallel": [ | ||
| "F7G8H9", | ||
| ... | ||
| ], | ||
| "sessions_after": [ | ||
| "I0J1K2", | ||
| ... | ||
| ], | ||
| "sessions_before": [ | ||
| "L3M4N5", | ||
| ... | ||
| ], | ||
| "next_session": "O6P7Q8", | ||
| "prev_session": "R9S0T1" | ||
| }, | ||
| } | ||
| ``` | ||
| </details> | ||
|
|
||
| | ||
|
|
||
| The fields are as follows: | ||
|
|
||
| | Key | Type | Notes | | ||
| |------------------------|-------------------------------------------|---------------------------------------------------------------| | ||
| | `code` | `string` | Unique identifier for the session | | ||
| | `title` | `string` | Title of the session | | ||
| | `speakers` | `array[string]` | List of codes of the speakers | | ||
| | `session_type` | `string` | Type of the session (e.g. Talk, Workshop, Poster, etc.) | | ||
| | `slug` | `string` | URL-friendly version of the title | | ||
| | `track` | `string` \| `null` | Track of the session (e.g. PyData, Web, etc.) | | ||
| | `abstract` | `string` | Abstract of the session | | ||
| | `tweet` | `string` | Tweet-length description of the session | | ||
| | `duration` | `string` | Duration of the session in minutes | | ||
| | `level` | `string` | Level of the session (e.g. beginner, intermediate, advanced) | | ||
| | `delivery` | `string` | Delivery mode of the session (e.g. in-person, remote) | | ||
| | `resources` | `array[object[string, string]]` \| `null` | List of resources for the session: `{"resource": <url>, "description": <description>}` | | ||
| | `room` | `string` \| `null` | Room where the session will be held | | ||
| | `start` | `string (datetime ISO format)` \| `null` | Start time of the session | | ||
| | `end` | `string (datetime ISO format)` \| `null` | End time of the session | | ||
| | `website_url` | `string` | URL of the session on the conference website | | ||
| | `sessions_in_parallel` | `array[string]` \| `null` | List of codes of sessions happening in parallel | | ||
| | `sessions_after` | `array[string]` \| `null` | List of codes of sessions happening after this session | | ||
| | `sessions_before` | `array[string]` \| `null` | List of codes of sessions happening before this session | | ||
| | `next_session` | `string` \| `null` | Code of the next session in the same room | | ||
| | `prev_session` | `string` \| `null` | Code of the previous session in the same room | | ||
|
|
||
| | ||
|
|
||
| ## `speakers.json` | ||
|
|
||
| <details> | ||
| <summary>Example speaker data JSON</summary> | ||
|
|
||
| ```json | ||
| { | ||
| "B4D5E6": { | ||
| "code": "B4D5E6", | ||
| "name": "A Speaker", | ||
| "biography": "Some bio", | ||
| "avatar": "https://pretalx.com/media/avatars/picture.jpg", | ||
| "slug": "a-speaker", | ||
| "submissions": [ | ||
| "A1B2C3", | ||
| ... | ||
| ], | ||
| "affiliation": "A Company", | ||
| "homepage": "https://example.com", | ||
| "gitx": "https://github.com/B4D5E6", | ||
| "linkedin_url": "https://www.linkedin.com/in/B4D5E6", | ||
| "mastodon_url": "https://mastodon.social/@B4D5E6", | ||
| "twitter_url": "https://x.com/B4D5E6" | ||
| }, | ||
| ... | ||
| } | ||
| ``` | ||
| </details> | ||
|
|
||
| | ||
|
|
||
| The fields are as follows: | ||
|
|
||
| | Key | Type | Notes | | ||
| |----------------|--------------------|-----------------------------------------------------------------------| | ||
| | `code` | `string` | Unique identifier for the speaker | | ||
| | `name` | `string` | Name of the speaker | | ||
| | `biography` | `string` \| `null` | Biography of the speaker | | ||
| | `avatar` | `string` | URL of the speaker's avatar | | ||
| | `slug` | `string` | URL-friendly version of the name | | ||
| | `submissions` | `array[string]` | List of codes of the sessions the speaker is speaking at | | ||
| | `affiliation` | `string` \| `null` | Affiliation of the speaker | | ||
| | `homepage` | `string` \| `null` | URL/text of the speaker's homepage | | ||
| | `gitx` | `string` \| `null` | URL/text of the speaker's GitHub/GitLab/etc. profile | | ||
| | `linkedin_url` | `string` \| `null` | URL of the speaker's LinkedIn profile | | ||
| | `twitter_url` | `string` \| `null` | URL of the speaker's Twitter profile | | ||
| | `mastodon_url` | `string` \| `null` | URL of the speaker's Mastodon profile | | ||
| | `website_url` | `string` | URL of the speaker's profile on the conference website | |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| [tool.isort] | ||
| profile = "black" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,5 +4,6 @@ pre-commit | |
|
|
||
| requests | ||
| pydantic | ||
| python-dotenv | ||
| python-slugify | ||
| tqdm | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.