This issue is intended to collect and discuss ideas about Pico 2.0 3.0 4.0, especially on how we can make Pico's plugin system more flexible than ever. The changes are very substantial and far reaching, therefore they can't make it into the soon-to-be-released Pico 2.0 3.0. The changes break BC, therefore they can't make it into Pico 1.1 2.1 3.1, but Pico 2.0 3.0 4.0. Features planned for Pico 2.0 can be found in #270. Feedback is appreciated! 😃

General

• Move Pico to the \picocms\Pico namespace and use a PSR-4 autoloader. Use PicoDeprecated to also provide all classes in the root namespace (using class_alias()), otherwise this would break all existing plugins.
• Add Unit Tests.
• Split Pico::run() into multiple public methods ("phases"), but still call them through Pico::run() (init, request URL, loading contents, evaluating contents (YAML + Markdown), page discovery, page rendering (Twig)).
• Use PicoMarkdownParser, PicoYamlParser and PicoTemplateEngine instead of \Parsedown, \Symfony\Component\Yaml\Parser and \Twig_Environment as type hints and add appropriate wrapper classes for Parsedown, Symfony YAML and Twig, allowing one to extend/replace them completely.
  • Enable some useful Symfony YAML flags by default (like converting keys to string automatically).
  • Also see cancelled "replace YAML/Parsedown/Twig" ToDo below.
• Drop all remaining pre-v1.0 behavior from PicoDeprecated, also remove v1.0+ behaviour with a notable performance impact
• Think about how users can easily install plugins with dependencies when they aren't using composer, but a pre-bundled release
• Think about refactoring Pico's plugin discovery process in general: having a lot of (partially duplicate) code for filesystem-based and composer-based plugins is bad, furthermore there are issues with the current system like Error running Pico on Synology Diskstation #538; maybe use extensible iterators?
• Allow theme developers to register meta headers and change Twig's default config (maybe using a config.yml in the theme's dir?) Implemented with Pico 2.1
• Allow theme developers to register static pages in Pico's pico-theme.yml (e.g. a virtual contact page is added to the pages list not requiring a contact.md with e.g. Template: contact to exist)
• Support relative page URLs in Markdown files
  • We must likely hook into Parsedown and make them absolute
  • Relative paths to other files pages (e.g. assets/image.jpg) should be supported, too
  • Relative paths must be interpreted so that content dir breakouts are stripped, i.e. ../page from content/sub/page.md ends up as https://example.com/pico/page, whereas ../../assets/index.jpg ends up as https://example.com/pico/assets/index.jpg (instead of https://example.com/assets/index.jpg)

Page management

• Implement lazy loading using page objects (i.e. use simple objects instead of $page arrays)
  • Performance, Performance, Performance
  • Implement ArrayAccess; not only to maintain BC, but also to make the object's data easier to use (especially for non-experienced developers)
    • Also switch a page's meta data to ArrayAccess objects and allow case insensitive keys (i.e. it no longer matters whether a meta value is Title or title, one can always access it with Title, title, or even tItLe)
    • Only propagate title, date, template and hidden as page data; everything else (e.g. description, author, robots) gets ordinary YAML metadata and can be accessed via page.meta
  • Pico initially loads no file contents, only page.id and page.url are available at first; as soon as something else is accessed, the file's contents are read; depending on what has been accessed, Pico processes either the YAML frontmatter or the Markdown contents (i.e. $page['content'] won't be deprecated anymore).
  • Plugin developers should be able to implement dynamic values with callbacks (e.g. PicoPage::addDynamicValue(string $key, callable $callback)). This also allows plugin developers to implement lazy loading for custom values.
  • Open question: Should we still allow regular page arrays in the $pages array? Probably not: plugin developers will have to differentiate both cases otherwise, what makes the whole feature a pain in the ass for them. Thus we need a conversion method, otherwise we would break all existing plugins...
  • Page objects don't know their respective next and previous pages, but parent and child pages (see below)
• Implement a lazy page tree (using iterable objects) as better performing alternative to the regular pages array
  • Performance, Performance, Performance!
    • Rather than always loading the whole page tree (a Traversable, ArrayAccess object like the pages array, see above), nothing is loaded by default.
    • When accessing the page tree for the first time, Pico discovers only pages directly inside content/ (with the second level of lazy loading as elucidated above). Pages in sub-directories are accessible through a children key. However, the contents of directories aren't discovered until they are explicitly requested by accessing said children key.
    • Most themes build their page menu by iterating over pages on the first level anyway. This allows Pico to discover only the pages it needs to know, i.e. just the pages directly inside the content/ dir and without any sub directory. This should heavily improve performance when a Pico instance is supposed to serve hundreds or thousands of pages.
    • Even the regular pages array is actually empty in the beginning. However, by accessing the variable (i.e. by iterating over it or by accessing a key) the whole page tree is being loaded (what shatters our efforts). We should encourage plugin/theme developers to use the page tree instead.
  • Pico manages a global sorted page tree
    • Drop support for sorting the pages list globally and rather use concomitant alphabetical sorting; i.e. Pico first discovers content/index.md only, then all other pages in content (e.g. content/foo.md) and sorts them alphabetically, then checks for a content/foo/ directory and discovers all pages in it (e.g. content/foo/bar.md), etc. and slices the just discovered pages in
  • Drop support for the $pages variable and force users to use the pages() function
    • Open question: PicoDeprecated should implement it as some magic variable to still support lazy loading (is this even possible?)
    • The pages() function should return an object instead of an array with sortable page objects wrapping the requested page objects; the pages list object can then be sorted and set a page's respective previous and next page to the page wrapper
      • It should be possible to perform both shallow and recursive sorting
  • Open question: How do we make this backwards-compatible with PicoDeprecated?
    • Use ArrayObject or ArrayIterator instead of a regular $pages array. This allows us to convert page arrays to objects as soon as they are added.
    • However, PHP's built-in array functions (e.g. array_keys()) won't work anymore... According to that we must pass a regular array to the onPagesLoaded event for older plugins, otherwise we would pretty likely break many of them. PicoDeprecated could then iterate through the $pages array and convert them appropriately.
    • Use event system versioning?
  • Refactor the pages() function to use offset and length parameters instead of depth, depthOffset and offset; see Show (grand)parent-page title #627 (comment) and earlier comments
    • Utilize theme API versioning to actually drop depth and depthOffset, but use PicoDeprecated to inject the old function for old themes
    • Thinking about using API versioning anyway, we could also rename length to depth - it better reflects what it does... But it's not to confuse with the "old" depth
    • Taking page objects into consideration we should use those in most cases, i.e. there should be keys to specifically not just access a page's child pages, but ancestors until a user-defined depth

Plugin event system

• Don't trigger all events on all plugins. Let plugins register the events they want to use instead. This heavily increases performance with a large number of plugins, because method_exists calls are comparatively expensive compared to a simple foreach per event.
  • Open question: Either introduce a new onSinglePluginLoaded event or a new PicoPluginInterface::getEvents() method.
    • The latter breaks BC, but with this new approach, we must refactor AbstractPicoPlugin::handleEvent() anyway, therefore we probably can't implement it without breaking BC one way or the other.
    • We can circumvent this by letting \AbstractPicoPlugin and \picocms\Pico\AbstractPlugin differ in functionality (i.e. the first mentioned implements the getEvents() method in a BC way by examining a ReflectionClass of the plugin, or by letting \PicoPluginInterface lack the getEvents() method entirely).
• Allow plugins to return false on preliminary events (e.g. onContentLoading) to prevent Pico from performing a specific processing step (Pico::run() skips Pico::loadFileContent()). Returning true or null works as with Pico 1.0 and changes nothing. The subsequent event is still triggered (onContentLoaded), but the payload variable ($rawContent) is empty. The event is triggered with special priority on this plugin (regardless of the regular processing order), so it can set the variable before any other plugin receives the event.
  • Example:
    • A markdown cache plugin returns false during onMetaParsing and onContentParsing to load both meta data and the parsed contents from its cache.
  • Affected Events:
    • onContentLoading (completely skips on404Content… events) and onContentLoaded
    • on404ContentLoading and on404ContentLoaded
    • onMetaParsing and onMetaParsed
    • onContentParsing (simulates onContentPrepared), onContentPrepared and onContentParsed
    • onPagesLoading (onSinglePage… events will be simulated) and onPagesLoaded
    • onSinglePageLoading (new event) and onSinglePageLoaded
    • onPageRendering and onPageRendered
• Allow plugins to return false on the onRequestUrl or onRequestFile events to completely skip Pico's processing. The only remaining event to trigger is onOutput (new event that is only triggered when Pico's processing is skipped) right before Pico returns $output.
  • Example:
    • A static HTML cache plugin returns false during onRequestFile, bypasses Pico's processing completely and returns the cached contents during onOutput.

New official plugins

• Markdown cache
• Static HTML cache
  • Save rendered output of pages to static HTML files
  • Rely on OS to detect file changes (last modification time of .md files)
  • Add Cache: No meta header to prevent pages from being cached
  • Add a event to let plugins "register" non-content pages for caching (should be triggered right after onConfigLoaded to allow plugins to change their behavior when caching is requested)
  • Explicitly allow combining statically cached and dynamic pages
  • Open questions
    • What happens when a page is added (i.e. page navigation changes)?
    • What happens when a plugin or theme is added/updated/removed?
    • How to determine all URLs that need to be parsed? Markdown files don't necessarily have a 1:1 relation to pages, just think of collections or hidden meta files
      • Ignore files and directories starting with a _
      • Allow users to explicitly specify the URL of a page
  • Use this feature to allow Pico to act as a static website generator (allow plugins to distinct between "static HTML cache" and "static website generator" mode)
    • Use Pico (with Travis, PHP's development server and wget -r) for our website rather than Jekyll
  • Plugin plugins (:smile:):
    • Add a URLs sequence meta header to support alternative URLs (like Jekyll's Redirect From plugin
• Search (using Lucene?)
  • Problem: How to determine the URL of a found Markdown file? Markdown files don't necessarily have a 1:1 relation to pages, just think of collections or hidden meta files
  • Possible solution: Use a static HTML cache and search in the HTML files
  • Possible solution: Do the exact same things as the static HTML cache (see above)
• Multilanguage (i18n)
  • Contents: use language-specific content directories or file extensions?
  • Themes: use PHP intl and Twig_Extensions_Extension_Intl
  • References: How to create multilingual web site? #335
• Performance statistics: See Pico 4.0 and beyond #317 (comment)
• Import plugins to import contents of other CMS (e.g. WordPress, see https://github.com/gilbitron/WordPress-to-Pico-Exporter)
  • Use a HTML to Markdown converter?
• Data Files
  • Support independent (meta) data files (e.g. content/catalog.yml)
  • The files are accessed similar to pages (e.g. {{ data.catalog }}).
  • A data file named after a markdown file (e.g. both content/catalog.yml and content/catalog.md exist) is non-recursively merged into the page's meta data (i.e. into {{ pages.catalog.meta }}). However, the YAML frontmatter takes preference and the data file can still be accessed via {{ data.catalog }}.
  • The same happens for all pages (non-recursive) in a directory if there's a data file with the same name as the directory (e.g. _collection.yml and _collection/ directory). You can enforce recursion for e.g. _collection/subdir/ by creating _collection/subdir.yml.
• Redirect pages (like Jekyll's Redirect From plugin)

Not planned anymore

• Allow a single plugin to hook into Pico to basically replace YAML/Parsedown/Twig with something different. Rather than hooking into the instantiation of \Symfony\Component\Yaml\Parser in Pico::parseFileMeta(), Pico::registerParsedown() and/or Pico::registerTwig(), it should be possible to replace the Pico::parseFileMeta() method, the Pico::prepareFileContent()/Pico::parseFileContent() methods (+ the markdown Twig filter in PicoTwigExtension) and/or the call of Pico::$twig->render(). Otherwise the plugin needs to re-implement the internal structures and workings of the YAML parser/Parsedown/Twig, what isn't desirable. I'm currently not sure about how this interacts with the $twig parameter of the onPageRendering event (maybe drop the parameter and add a new onTwigRegistered event?). The plugin needs to be registered explicitly in config/config.php to work.
  • Example (quite a stretch):
    • Instead of parsing Markdown, parse MediaWiki syntax.