Robust, asynchronous project loading by jtulach · Pull Request #9680 · apache/netbeans

jtulach · 2026-10-07T15:32:00Z

this is work in progress to be integrated for after NetBeans 32 is branched
initial goal: use the same infrastructure for lazy project opening when switching groups
- requires bunch of refactorings of the OpenProjectList
- desire: use the same infrastructure when opening projects in VSCode extension too
subsequent goal: clean it up and make it more robust
- possibly we could remove @RandolyFailing from the tests in there
- remove enter/exit
it may require some time for stabilization

…ojects with TestProjectOpenedHookImpl

…st up-to-date array of projects

…sLoading

jtulach · 2026-10-08T05:48:23Z

+
+    public boolean waitFinished(long toMillis);
+
+    public void enter();


These enter and exit methods do not fit into the intended workflow.

they should be removed and replaced by ...

a long running operations in a dedicated request processor thread

eppleton

Thanks for working on this. I reviewed 0674e5c and checked the points inline locally against master. The main ones are the openProjects() future contract, startup events firing under OpenProjectList.MUTEX, a deadlock in replaceProjects, and a group selected before load opening the old projects. A few test-helper issues in OpenProjectListTest hide failures.

eppleton · 2026-10-08T07:55:09Z

@@ -232,7 +210,37 @@ static void preferredProject(final Project lazyP) {
    }

    public Future<Project[]> openProjectsAPI() {


This Future no longer waits for an open()/close() that is in progress. On master get() also waited until entered == 0, which is what the OpenProjects.openProjects() javadoc promises ("awaits for current modifications to finish"). isDone() still checks entered, so it can return false while get() returns immediately, and enteredZeroed is now signalled but never awaited.

With a project whose open hook blocks, master's get(300ms) throws TimeoutException; this branch returns the list. OpenProjectListTest.testProjectOpenedClosed and testProjectClosedRace fail on every isolated run here and pass on master. CI won't show it, because the class is @RandomlyFails and CI sets ignore.random.failures=true.

I see the plan to remove enter/exit. Whatever replaces them needs to keep this contract, and dropping @RandomlyFails from OpenProjectListTest would let CI guard it.

eppleton · 2026-10-08T07:55:09Z

+            mainProject = unwrapProject(mainProject);
+            getRecentTemplates().addAll(recentTemplates);
+
+            MUTEX.postReadRequest(() -> {


updateGlobalState is called inside writeAccess, so postReadRequest is delayed until the write lock is released and then runs in read access. As a result, PROPERTY_OPEN_PROJECTS / PROPERTY_MAIN_PROJECT at startup are delivered with OpenProjectList.MUTEX held; on master they fired with no lock. A listener on this branch sees MUTEX.isReadAccess() == true; on master it is false.

All OpenProjects listeners in the IDE get these events via OpenProjectsTrampolineImpl, so they now run under the lock the class comment says must never be held while reaching ProjectManager.MUTEX. Any listener needing write access (e.g. setMainProject) hits the read→write upgrade error. Could the callback be split into "apply state" (under the lock) and "fire" (after writeAccess returns), as before?

eppleton · 2026-10-08T07:55:09Z

+
+    private final void replaceProjectsImpl(List<LazyProject> projects, URL mainProject) {
+        if (openProjects != null) {
+            close(openProjects.toArray(Project[]::new), false);


This deadlocks when replaceProjects is called while the startup load is still running. replaceProjects holds MUTEX.writeAccess and calls close(). LOAD.closeBeforeOpen returns false for a project the loader has already dequeued, so close() calls LOAD.waitFinished(0). The loader then blocks on MUTEX (notifyOpened → prepareTemplates → getDefault() → readAccess). I reproduced it locally: both threads hang permanently.

Independent of the deadlock, this runs close() (project saves, property changes, closeAllDocuments) under MUTEX. The class comment warns against reaching ProjectManager.MUTEX from there.

eppleton · 2026-10-08T07:55:09Z

            }
        };
-        pchSupport = new PropertyChangeSupport( this );
+        replaceProjectsImpl(loadProjectList(), main);


Capturing the URLs here, in the constructor, changes behaviour when a group is selected before projects are loaded (e.g. --open-group at startup). Group.setActiveGroup with projectsLoaded == false first calls OpenProjects.getDefault().getOpenProjects(), which creates OpenProjectList and captures the old URLs. Only afterwards does it write the new group's URLs with setOpenProjectsURLsAsStrings. On master loadInBackground read the URLs when it ran, so it picked up the new group.

With saved settings [a] and a group {b} activated before load, master opens b and this branch opens a.

eppleton · 2026-10-08T07:55:09Z

        Iterator<ExtIcon> iconIt = icons.iterator();
-        
+
        while(urlIt.hasNext() && namesIt.hasNext() && iconIt.hasNext()) {


The projects opened at startup now come from this zip. A URL without a matching name and icon is silently skipped, whereas on master the URL list alone decided what was opened. Group.java:187 writes only the URL list, so the three lists can get out of step.

eppleton · 2026-10-08T07:55:09Z

+
+        void removeOpenFile(FileObject fo) {
+            var previousValue = openFiles.remove(fo.toURL().toExternalForm());
+            assertNotNull("There should be previous value for " + fo);


assertNotNull(String) only checks the message string, so this always passes; previousValue is never used.

eppleton · 2026-10-08T07:55:09Z

-        public Set<String> openFiles = new HashSet<String>();
-        public Map<Project,Set<String>> urls4project = new HashMap<Project,Set<String>>();
+    public static final class TestOpenCloseProjectDocument implements ProjectUtilities.OpenCloseProjectDocument {
+        private final Map<String, Exception> openFiles = new HashMap<>();


openFiles is written from OPENING_RP (via ProjectUtilities.openProjectFiles) and read from the test thread. Maybe use a synchronized or concurrent map?

eppleton · 2026-10-08T07:55:09Z

        }
-        assertFalse ("Document f1_1_open isn't loaded.", handler.openFiles.contains (f1_1_open.toURL ().toExternalForm ()));
-        assertFalse ("Document f1_2_open isn't loaded.", handler.openFiles.contains (f1_2_open.toURL ().toExternalForm ()));
+        handler.assertNotOpened("Document f1_1_open is loaded.",f1_1_open);


Nit: "is loaded." on an assertNotOpened. The same applies to the next line.

eppleton · 2026-10-08T07:55:10Z

+        });
+    }
+
+    @NbBundle.Messages(value = {"#NOI18N", "LOAD_PROJECTS_ON_START=true"})


Nit: LOAD_PROJECTS_ON_START is now used only in OpenProjectList.loadProjectList, so the @Messages declaration could move there.

eppleton · 2026-10-08T07:55:10Z

+ * <p>
+ * <h3>How it Should Work?</h3>
+ *
+ * When there is a needed to change the list of opened projects, then let's


Nit, javadoc typos: "When there is a needed to change" here, "currated" (line 333), and a stray } at the end of the finishOpening @return (line 345).

neilcsmith-net · 2026-10-08T09:01:58Z

Hopefully this work might address an occasionally seen issue where the loading gets "stuck" completely. More often in my platform IDE than NetBeans, although it does show there. See my comment on a recently reported issue at #9657 (comment) The project gets stuck as LazyProject. I have a log output (on FINE) that suggests that the real project is created, then replaced by the lazy project again. Never could work out the trigger.

jtulach added 5 commits October 7, 2026 15:43

Separating OpenProjectLoading logic into its own class

24a62f9

Making the ProjectsRootNodePhysicalViewTest more robust by holding pr…

4a83c82

…ojects with TestProjectOpenedHookImpl

OpenProjectList.openProjects future must wait, but then return the mo…

210017b

…st up-to-date array of projects

Recent templates shall not be handled by OpenProjectsLoading

9160302

No references to OpenProjectListSettings.getInstance() in OpenProject…

48cd059

…sLoading

jtulach added this to the NB33 milestone Oct 7, 2026

jtulach requested review from MartinBalin, eppleton and mbien October 7, 2026 15:32

jtulach self-assigned this Oct 7, 2026

jtulach added do not merge Don't merge this PR, it is not ready or just demonstration purposes. Project UI View labels Oct 7, 2026

jtulach added 4 commits October 7, 2026 18:03

Extracting communication interface to OpenProjectsLoading task

2012809

Creating a dedicated OpenProjectOperation

5732822

Making OpenProjectListTest more reliable

195dffb

Request projects to open when creating OpenProjectsLoading

cd28345

jtulach commented Oct 8, 2026

View reviewed changes

jtulach added 2 commits October 8, 2026 08:31

Make sure the erasured type is Future

b2fcfde

Allow replacing projects

0674e5c

eppleton reviewed Oct 8, 2026

View reviewed changes

jtulach added 3 commits October 8, 2026 09:56

Allow multiple calls into replaceProjects

89d3f90

Describing intended implementation

db6c7d9

Moving comparator to its own class

5a27f59

Can every problem be solved by another layer of indirection?

b14f1e9

Navigation Menu

Repository navigation

Robust, asynchronous project loading - #9680

Robust, asynchronous project loading#9680
jtulach wants to merge 15 commits into
apache:masterfrom
jtulach:jtulach/OpenProjectLoading

jtulach commented Oct 7, 2026 •

edited

Loading

Uh oh!

jtulach Oct 8, 2026

Uh oh!

eppleton left a comment

Uh oh!

eppleton Oct 8, 2026

Uh oh!

eppleton Oct 8, 2026

Uh oh!

eppleton Oct 8, 2026

Uh oh!

eppleton Oct 8, 2026

Uh oh!

eppleton Oct 8, 2026

Uh oh!

eppleton Oct 8, 2026

Uh oh!

eppleton Oct 8, 2026

Uh oh!

eppleton Oct 8, 2026

Uh oh!

eppleton Oct 8, 2026

Uh oh!

eppleton Oct 8, 2026

Uh oh!

neilcsmith-net commented Oct 8, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants


		public boolean waitFinished(long toMillis);

		public void enter();

		@@ -232,7 +210,37 @@ static void preferredProject(final Project lazyP) {
		}

		public Future<Project[]> openProjectsAPI() {

		Iterator<ExtIcon> iconIt = icons.iterator();


		while(urlIt.hasNext() && namesIt.hasNext() && iconIt.hasNext()) {

Conversation

jtulach commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

eppleton left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

neilcsmith-net commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jtulach commented Oct 7, 2026 •

edited

Loading

neilcsmith-net commented Oct 8, 2026 •

edited

Loading