zookeeper

Zookeeper is not running

Env: Hortonworks 2.1

Error: connection refused error:

Reason: 

  1. It’s a snapshot that started with nodes /A/B/C present. Before it got to B, both B and C got removed. So, the snapshot contains only /A, but the txn log starts with “delete C”. When it replays the log, it tries to increment the cversion of the parent, which is /A/B. It isn’t present in the snapshot and the recovery crashes.
  2. The log space is full

Solution:

  1. Remove the /var/zookeeper/version-2 solves the problem

 

Advertisements
environment basics

List of HADOOP port numbers

List of port numbers: ref link
NAMENODE PORT: 50070 => dfs.http.address
RESOURCE MANAGER PORT : 8088 => yarn.resourcemanager.webapp.address
JOB HISTORY PORT : 19888 => mapreduce.jobhistory.webapp.address
DATANODE PORT: 50075 => dfs.datanode.http.address
DATANODE TCP PORT: 8042
SECND. NAMENODE PORT: 50090 => dfs.secondary.http.address
CHK. POINT PORT: 50105 => dfs.backup.http.address

All Hadoop Daemons:
/logs
Exposes, for downloading, log files in the Java system property hadoop.log.dir.
/logLevel
Allows you to dial up or down log4j logging levels. This is similar to hadoop daemonlog on the command line.
/stacks
Stack traces for all threads. Useful for debugging.
/metrics
Metrics for the server. Use /metrics?format=json to retrieve the data in a structured form. Available in 0.21.

Namenode:

<namenode IP>:50070

/
Shows information about the namenode as well as the HDFS. There’s a link from here to browse the filesystem, as well.
/dfsnodelist.jsp?whatNodes=(DEAD|LIVE)
Shows lists of nodes that are disconnected from (DEAD) or connected to (LIVE) the namenode.
/fsck
Runs the “fsck” command. Not recommended on a busy cluster.
/listPaths
Returns an XML-formatted directory listing. This is useful if you wish (for example) to poll HDFS to see if a file exists. The URL can include a path (e.g., /listPaths/user/philip) and can take optional GET arguments: /listPaths?recursive=yes will return all files on the file system;/listPaths/user/philip?filter=s.* will return all files in the home directory that start withs; and /listPaths/user/philip?exclude=.txt will return all files except text files in the home directory. Beware that filter and exclude operate on the directory listed in the URL, and they ignore the recursive flag.
/data and /fileChecksum
These forward your HTTP request to an appropriate datanode, which in turn returns the data or the checksum.

Datanodes :

<datanode IP>:50075

/browseBlock.jsp, /browseDirectory.jsp, tail.jsp, /streamFile,/getFileChecksum
These are the endpoints that the namenode redirects to when you are browsing filesystem content. You probably wouldn’t use these directly, but this is what’s going on underneath.
/blockScannerReport
Every datanode verifies its blocks at configurable intervals. This endpoint provides a listing of that check.

The secondarynamenode exposes a simple status page with information including which namenode it’s talking to, when the last checkpoint was, how big it was, and which directories it’s using.

 

Under the Covers for the Developer and the System Administrator

Internally, Hadoop mostly uses Hadoop IPC to communicate amongst servers. (Part of the goal of the Apache Avro project is to replace Hadoop IPC with something that is easier to evolve and more language-agnostic;HADOOP-6170 is the relevant ticket.) Hadoop also uses HTTP (for the secondarynamenode communicating with the namenode and for the tasktrackers serving map outputs to the reducers) and a raw network socket protocol (for datanodes copying around data).

The following table presents the ports and protocols (including the relevant Java class) that Hadoop uses. This table does not include the HTTP ports mentioned above.

Daemon Default Port Configuration Parameter Protocol Used for
Namenode 8020 fs.default.name? IPC: ClientProtocol Filesystem metadata operations.
Datanode 50010 dfs.datanode.address Custom Hadoop Xceiver: DataNode andDFSClient DFS data transfer
Datanode 50020 dfs.datanode.ipc.address IPC: InterDatanodeProtocol,ClientDatanodeProtocol
ClientProtocol
Block metadata operations and recovery
Backupnode 50100 dfs.backup.address Same as namenode HDFS Metadata Operations
Jobtracker Ill-defined.? mapred.job.tracker IPC: JobSubmissionProtocol,InterTrackerProtocol Job submission, task tracker heartbeats.
Tasktracker 127.0.0.1:0¤ mapred.task.tracker.report.address IPC: TaskUmbilicalProtocol Communicating with child jobs
? This is the port part of hdfs://host:8020/.
? Default is not well-defined. Common values are 8021, 9001, or 8012. See MAPREDUCE-566.
¤ Binds to an unused local port.
environment basics

where is HADOOP configuration files

Configuration files can be found at /etc/hadoop/conf/

  • hadoop-env.sh

holds the environment variables used in hadoop

  • yarn-env.sh

holds the environment variables used in YARN

  • mapred-env.sh

holds the environment variables for MAPREDUCE

  •  core-site.xml

settings of IO common to HDFS and MAPREDUCE

  • hdfs-site.xml

settings for datanode,hdfs daemons, namenode, secondary namenode

  • mapred-site.xml

settings for hadoop mapreduce

  • yarn-site.xml

settings for resource manager and node manager

  • master

machine names of secondary namenode

  • slave

machine names of data nodes

  •  capacity-scheduler.xml

holds the scheduler properties of yarn

  • hadoop-policy.xml

security and acl properties of the cluster and data nodes

  • container-executor.cfg

Holds log configuration for node managers

HDFS

HDFS FEDERATION

Created to increase the namespace and Namenode load. If found that the number of files are growing, then the number of namespace will also grow and the same will grow the name space.

To federate the directories. say /finance is in one namenode and /hr is in another namenode. data node stores the data of both. The block pool managed for each namespace.

How to do it will be done on next post.