org.apache.hadoop.util

Class MRAsyncDiskService

java.lang.Object

org.apache.hadoop.util.MRAsyncDiskService

public class MRAsyncDiskService
extends Object

This class is a container of multiple thread pools, each for a volume, so that we can schedule async disk operations easily. Examples of async disk operations are deletion of files. We can move the files to a "toBeDeleted" folder before asychronously deleting it, to make sure the caller can run it faster. Users should not write files into the "toBeDeleted" folder, otherwise the files can be gone any time we restart the MRAsyncDiskService. This class also contains all operations that will be performed by the thread pools.

Field Summary

Fields

Modifier and Type	Field and Description
static org.apache.commons.logging.Log	LOG
static String	TOBEDELETED

Constructor Summary

Constructors

Constructor and Description
MRAsyncDiskService(org.apache.hadoop.fs.FileSystem localFileSystem, String[] nonCanonicalVols)
MRAsyncDiskService(org.apache.hadoop.fs.FileSystem localFileSystem, TaskController taskController, String[] nonCanonicalVols) Create a AsyncDiskServices with a set of volumes (specified by their root directories).
MRAsyncDiskService(JobConf conf) Initialize MRAsyncDiskService based on conf.

Method Summary

Methods

Modifier and Type	Method and Description
boolean	awaitTermination(long milliseconds) Wait for the termination of the thread pools.
void	cleanupAllVolumes() Move all files/directories inside volume into TOBEDELETED, and then delete them.
void	cleanupDirsInAllVolumes(String[] dirsToDelete) Move specified directories/files in each volume into TOBEDELETED, and then delete them.
boolean	moveAndDeleteAbsolutePath(String absolutePathName) Move the path name to a temporary location and then delete it.
boolean	moveAndDeleteFromEachVolume(String pathName) Move the path name on each volume to a temporary location and then delete them.
boolean	moveAndDeleteRelativePath(String volume, String pathName) Move the path name on one volume to a temporary location and then delete them.
void	shutdown() Gracefully start the shut down of all ThreadPools.
List<Runnable>	shutdownNow() Shut down all ThreadPools immediately.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

LOG

public static final org.apache.commons.logging.Log LOG

TOBEDELETED

public static final String TOBEDELETED

See Also:

Constant Field Values

Constructor Detail

MRAsyncDiskService

public MRAsyncDiskService(org.apache.hadoop.fs.FileSystem localFileSystem,
                  TaskController taskController,
                  String[] nonCanonicalVols)
                   throws IOException

Create a AsyncDiskServices with a set of volumes (specified by their root directories). The AsyncDiskServices uses one ThreadPool per volume to do the async disk operations.

Parameters:

localFileSystem - The localFileSystem used for deletions.

nonCanonicalVols - The roots of the file system volumes, which may be absolte paths, or paths relative to the ${user.dir} system property ("cwd").

Throws:

IOException

MRAsyncDiskService

public MRAsyncDiskService(org.apache.hadoop.fs.FileSystem localFileSystem,
                  String[] nonCanonicalVols)
                   throws IOException

Throws:

IOException

MRAsyncDiskService

public MRAsyncDiskService(JobConf conf)
                   throws IOException

Initialize MRAsyncDiskService based on conf.

Parameters:

conf - local file system and local dirs will be read from conf

Throws:

IOException

Method Detail

shutdown

public void shutdown()

Gracefully start the shut down of all ThreadPools.

shutdownNow

public List<Runnable> shutdownNow()

Shut down all ThreadPools immediately.

awaitTermination

public boolean awaitTermination(long milliseconds)
                         throws InterruptedException

Wait for the termination of the thread pools.

Parameters:

milliseconds - The number of milliseconds to wait

Returns:

true if all thread pools are terminated within time limit

Throws:

InterruptedException

moveAndDeleteRelativePath

public boolean moveAndDeleteRelativePath(String volume,
                                String pathName)
                                  throws IOException

Move the path name on one volume to a temporary location and then delete them. This functions returns when the moves are done, but not necessarily all deletions are done. This is usually good enough because applications won't see the path name under the old name anyway after the move.

Parameters:

volume - The disk volume

pathName - The path name relative to volume root.

Returns:

false if the file is not found

Throws:

IOException - If the move failed

moveAndDeleteFromEachVolume

public boolean moveAndDeleteFromEachVolume(String pathName)
                                    throws IOException

Move the path name on each volume to a temporary location and then delete them. This functions returns when the moves are done, but not necessarily all deletions are done. This is usually good enough because applications won't see the path name under the old name anyway after the move.

Parameters:

pathName - The path name relative to each volume root

Returns:

false If any of the target pathName did not exist, note that the operation is still done on all volumes.

Throws:

IOException - If any of the move failed

cleanupDirsInAllVolumes

public void cleanupDirsInAllVolumes(String[] dirsToDelete)
                             throws IOException

Move specified directories/files in each volume into TOBEDELETED, and then delete them.

Parameters:

dirsToDelete -

Throws:

IOException

cleanupAllVolumes

public void cleanupAllVolumes()
                       throws IOException

Move all files/directories inside volume into TOBEDELETED, and then delete them. The TOBEDELETED directory itself is ignored.

Throws:

IOException

moveAndDeleteAbsolutePath

public boolean moveAndDeleteAbsolutePath(String absolutePathName)
                                  throws IOException

Move the path name to a temporary location and then delete it. Note that if there is no volume that contains this path, the path will stay as it is, and the function will return false. This functions returns when the moves are done, but not necessarily all deletions are done. This is usually good enough because applications won't see the path name under the old name anyway after the move.

Parameters:

absolutePathName - The path name from root "/"

Returns:

false if we are unable to move the path name

Throws:

IOException - If the move failed