mirror of
https://github.com/eclipse-cdt/cdt
synced 2025-07-04 15:45:25 +02:00
68 lines
4.9 KiB
HTML
68 lines
4.9 KiB
HTML
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
|
|
<html>
|
|
<head>
|
|
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">
|
|
<META HTTP-EQUIV="Content-Style-Type" CONTENT="text/css">
|
|
<meta name="copyright" content="Copyright (c) IBM Corporation and others 2006. This page is made available under license. For full details see the LEGAL in the documentation book that contains this page." >
|
|
<title>RSE DataStore Memory Management</title>
|
|
<link rel="stylesheet" type="text/css" HREF="../book.css">
|
|
</head>
|
|
|
|
<body>
|
|
<h1><a name="dataelements">Memory Management in DataStore</a></h1>
|
|
<p>
|
|
A Memory Management scheme has been implemented for the DataStore that significantly reduces the memory footprint of the
|
|
DataElements in high-use, long up-time situations. A problem that was identified was that in the DataStore, DataElements were
|
|
rarely being destroyed - only when the files they represented were themselves deleted. When directories in the file system were explored,
|
|
DataElements were being created to represent the files in the directories, but these DataElements were cached and never
|
|
removed from the cache. Therefore, the number of DataElements in the DataStore was always increasing. With no opportunity
|
|
to clear the cache or remove elements, the memory usage of the server continued to grow boundlessly.
|
|
</p>
|
|
<h2><a name="spirit">A solution - Spirit DataElements</a></h2>
|
|
<p>
|
|
The solution to the problem of an ever-growing set of DataElements was simple - provide a mechanism of removing "old"
|
|
DataElements and thus shrinking the set. Since server memory real-estate comes at a much higher premium, the focus here
|
|
is on server-side memory reduction. The assumption then, is that DataElements in the DataStore will always remain in
|
|
memory on the client, but that in the mirror-image DataStore that resides on the server, DataElements can be removed.
|
|
</p>
|
|
<p>
|
|
Formerly, the RSE ran under the assumption that the client and server DataStores mirrored each other. The new
|
|
implementation has a server DataStore tree that is only a subset of all the elements in the client tree (because
|
|
some elements get removed.) In order to accommodate this, a new boolean member variable was added to the DataElement
|
|
class called "isSpirit". When this variable is set to true it means different things on the client and server. On the
|
|
server, a "spirit" DataElement means the element is treated in much the same way as a "deleted" element. At the first
|
|
opportunity, the element is purged from the DataStore and garbage collected by the JVM - freeing up memory. On the client,
|
|
a "spirit" element means that that particular DataElement's counterpart has been made a spirit; thus the client "knows"
|
|
that its twin element on the server side has either been deleted, or is about to be deleted.
|
|
</p>
|
|
<h2><a name="disconnecting">Disconnecting "old" DataElements:</a></h2>
|
|
<p>
|
|
How is it determined when to mark a given DataElement as a spirit? It was decided that the decision to this would be
|
|
left to clients of the DataStore (the miners), rather than to the DataStore itself. This was done for the purposes of
|
|
granularity: some individual miners may not want DataElements to be ever purged, some might want only specific elements,
|
|
etc. As an example, the UniversalFileSystemMiner employs the FileClassifier to classify files returned from a directory
|
|
query, and after each file has been classified, the DataStore's disconnectObject() method is called on the DataElement
|
|
representing that file, setting the stage for its becoming a spirit.
|
|
</p>
|
|
<h2><a name="queue">Controlling the queue of DataElements:</a></h2>
|
|
<p>
|
|
A new class, the DataElementRemover, running in its own thread, maintains a queue of objects that were passed into
|
|
DataStore's disconnectObject() method; each object is stored in the queue along with the time it was added. The
|
|
DataElementRemover is configurable by two command-line options: -DSPIRIT_EXPIRY_TIME=x and -DSPIRIT_INTERVAL_TIME=y;
|
|
where x and y are integers representing a number of seconds. Every y seconds, the queue checks its elements and "makes
|
|
spirit" all those that are older than x seconds. The DataElement is then refreshed, and the change propagated to the
|
|
client. On the server side, the DataElement is deleted at the first opportunity.
|
|
</p>
|
|
<h2><a name="feature">Turning on the feature:</a></h2>
|
|
<p>
|
|
On the client side, this feature is always "on". It is the server which is configured to do or not to do the spirit
|
|
DataElement behaviour. This way, backwards compatibility is maintained - if a new client connects to an old server, or a
|
|
server with spirit turned off, the client detects this and operates as before. If an old client connects to a new server
|
|
with spirit turned on, the server detects that the client does not have spirit capability and behaves as it did before.
|
|
</p>
|
|
<p>
|
|
To turn on the spirit feature on the server side, one needs to include the command-line option -DDSTORE_SPIRIT_ON=true.
|
|
The server scripts have been packaged in order to do this by default.
|
|
</p>
|
|
</body>
|
|
</html>
|