Summary

A nice to have feature would be the ability to use views from other pipelines as inputs to a different pipeline.

Motivation

In large projects, multiple teams may use the same pipeline, and each team may have different production availability requirements. For example one team need high up time for there use of the pipeline, while another team is multiple times a day changing there views in pipeline, resulting i multiple down times for other teams.

Splitting a large pipeline into smaller, team-owned pipelines can help distribute responsibility, but it can lead to a lot of duplication if each pipeline needs the same base data.

Allowing pipelines to consume views from other pipelines would enable a modular approach, where teams can build on top of a base pipeline without duplicating logic.

Example usage:

CREATE TABLE base_view1 (
    ...
) WITH (
    'connectors' = '[{
        "transport": {
            "name": "feldera_pipline_input",
            "config": {
                "pipline_name": "base_pipline",
                "view": "base_view1",
            }
        }
    }]'
);

Reference-level explanation

This is the technical portion of the RFC. Explain the design in sufficient detail that:

• Its interaction with other features is clear.
• It is reasonably clear how the feature would be implemented.
• Corner cases are dissected by example.

Drawbacks

Why should we not do this?

Rationale and alternatives

• Why is this design the best in the space of possible designs?
• What other designs have been considered and what is the rationale for not choosing them?
• What is the impact of not doing this?

Prior art

Discuss prior art, both the good and the bad, in relation to this proposal.
A few examples of what this can include are:

• What lessons can we learn from what others have done here?
• Papers: Are there any published papers or great posts that discuss this?

Unresolved questions

• What related issues do you consider out of scope for this RFC that could be addressed in the future independently of the solution that comes out of this RFC?

Future possibilities

Think about what the natural extension and evolution of your proposal would
be and how it would affect the project as a whole in a holistic
way. This is also a good place to "dump ideas", if they are out of scope for the
RFC you are writing but otherwise related.