Activity

1. stiber

   added

   bugSomething isn't workingSomething isn't working

   urgent

   on Mar 14, 2025
2. stiber commented on Mar 14, 2025

   

   stiber

   on Mar 14, 2025

   ContributorAuthor

   More actions

   Note that in v0.9.3, which only serializes the synapse information, we are able to deserialize working synapses.
3. varndorfer commented on Mar 24, 2025

   

   varndorfer

   on Mar 24, 2025

   Contributor

   More actions

   The serialization file outputs the AllEdges.destinationVertices vector as all zeros. I wrote a hack to output the destVertexIndex_ vector in AllSynapsesDeviceFuncs_d.cpp at the end of a simulation and that was also all zero. The problem may be in AllSynapsesDeviceFuncs_d.cpp and not serialization
4. Hum-Bao commented on Mar 9, 2026

   

   Hum-Bao

   on Mar 9, 2026

   Contributor

   More actions

   From Copilot:

   Problem Summary
   In the GPU growth continuation path, destination vertex indices are ending up as zero in serialized state, which then breaks connectivity after deserialization and makes activity look mostly endogenous. The most likely failing path is not the destination assignment inside AllSynapsesDeviceFuncs_d.cpp:95, but host-device synchronization and serialization timing around ConnGrowth_d.cpp:83, AllSpikingSynapses_d.cpp:303, and Core.cpp:232.

   Assumptions and Scope
   Missing input: no exact repro config, no failing test output, and no captured serialized file excerpt were provided.
   I assumed this is a GPU run path (USE_GPU enabled), based on your reference to device files and GPU debug output.
   I focused on the growth path (ConnGrowth GPU update) and serialization/deserialization path, not static-only CPU behavior.
   I treated your observation of all-zero destination vectors as ground truth.
   Execution Trace
   Startup setup zeros edge vectors in AllEdges.cpp:104, then initializes initial connections.
   During GPU growth update, synapses are modified on device by updateSynapsesWeightsDevice called in ConnGrowth_d.cpp:67. Device-side create functions do write destination index (for example AllSynapsesDeviceFuncs_d.cpp:104).
   After the kernel, host is partially refreshed via ConnGrowth_d.cpp:83, which calls copyDeviceEdgeSumIdxToHost.
   That helper currently copies sourceVertexIndex and inUse but not destVertexIndex in AllSpikingSynapses_d.cpp:310 and AllSpikingSynapses_d.cpp:312.
   Serialization is invoked before finish in Core.cpp:232, while full copyFromGPU happens inside finish at GPUModel.cpp:125. So serialization can capture stale host edge fields.
   On deserialization, GPU path calls only gpuModel.copyCPUtoGPU at Serializer.cpp:70, and that function mainly updates edge index maps in GPUModel.cpp:283, not the full edge/vertex buffers unless operation-based copy methods are executed.
   Root Cause Analysis
   Most likely root cause:

   Host-side destination indices are not kept in sync from device during growth updates.
   Specifically, copyDeviceEdgeSumIdxToHost omits destVertexIndex copy in AllSpikingSynapses_d.cpp:303, while destination is mutated on device during add/create in AllSynapsesDeviceFuncs_d.cpp:432.
   Because serialization occurs before finish/copyFromGPU (Core.cpp:232), stale host values are written to file, often preserving zeros from initialization in AllEdges.cpp:104.
   Secondary hypothesis (also important):

   Deserialization for GPU only refreshes map data through Serializer.cpp:70 and GPUModel.cpp:283, so deserialized edge payloads may not fully reach device buffers.
   Why this matches your observation:

   It explains why destination data in serialized output can be all zeros even though device create functions appear correct.
   It also explains broken continuation behavior after deserialization.
   Proposed Fix
   Minimal targeted changes:

   Fix 1: copy destVertexIndex from device to host in copyDeviceEdgeSumIdxToHost.
   Fix 2: ensure host is synced from GPU before serialization.
   Fix 3: after deserialize on GPU, execute full copy-to-GPU operation chain, not map-only copy.
   Suggested patch-style snippet:

   
   void AllSpikingSynapses::copyDeviceEdgeSumIdxToHost(void *allEdgesDevice)
   {
      AllSpikingSynapsesDeviceProperties allEdgesDeviceProps;
      BGSIZE maxTotalSynapses = Simulator::getInstance().getMaxEdgesPerVertex()
                                * Simulator::getInstance().getTotalVertices();
   
      HANDLE_ERROR(cudaMemcpy(&allEdgesDeviceProps, allEdgesDevice,
                              sizeof(AllSpikingSynapsesDeviceProperties),
                              cudaMemcpyDeviceToHost));
   
      HANDLE_ERROR(cudaMemcpy(sourceVertexIndex_.data(), allEdgesDeviceProps.sourceVertexIndex_,
                              maxTotalSynapses * sizeof(int), cudaMemcpyDeviceToHost));
   
      // Keep destination indices synchronized for serialization and CPU-side index map rebuilds.
      HANDLE_ERROR(cudaMemcpy(destVertexIndex_.data(), allEdgesDeviceProps.destVertexIndex_,
                              maxTotalSynapses * sizeof(int), cudaMemcpyDeviceToHost));
   
      HANDLE_ERROR(cudaMemcpy(inUse_.data(), allEdgesDeviceProps.inUse_,
                              maxTotalSynapses * sizeof(unsigned char), cudaMemcpyDeviceToHost));
   }
   

   File: [Core.cpp](http://_vscodecontentref_/21)
   
   #if defined(USE_GPU)
      // Ensure host edge/vertex state is current before writing serialization file.
      OperationManager::getInstance().executeOperation(Operations::copyFromGPU);
   #endif
      serializer.serialize();
   
   File: [Serializer.cpp](http://_vscodecontentref_/22)
   
   #if defined(USE_GPU)
      // Push all deserialized host objects (vertices, edges, index maps) back to device.
      OperationManager::getInstance().executeOperation(Operations::copyToGPU);
   #endif
   

   Notes:

   The first change is the direct fix for missing destination synchronization.
   The second and third changes make serialization/deserialization behavior consistent for GPU continuation.
   Verification Steps
   Add a focused unit/integration check for GPU growth continuation path:
   Run a short GPU growth simulation that creates at least one non-self connection.
   Before serialization, assert there exists an in-use edge with non-zero destination index on host.
   Serialize, deserialize, then assert the same invariant again.
   Manual checks:
   Log a few in-use edges and print source/destination pairs right before serialization in Core.cpp:232.
   Confirm destination values are not uniformly zero in the serialized file.
   Continue simulation after deserialization and compare spike propagation against a non-deserialized control run.
   Existing commands:
   Run your standard unit/regression suite after patch.
   Add one regression test under RegressionTesting for serialize-deserialize-continue on GPU growth (if your test infra supports GPU in CI).
5. stiber

   assigned

   crobin5

   on Apr 21, 2026
6. stiber

   added

   serialization

   on Apr 21, 2026
7. stiber commented on Aug 21, 2026

   

   stiber

   on Aug 21, 2026

   ContributorAuthor

   More actions

   Closed automatically by cleanup script: resolved in merged pull request #942.
8. stiber

   closed this as completedon Aug 21, 2026