自定义Ranger服务

Ranger Stacks

Apache Ranger为Hadoop生态系统提供中心化的细粒度安全控制及审计。Ranger0.4版本支持HDFS、HBase、Hive、Knox、Storm等多种服务，自0.5版本后，增加了stack-model以支持用户自定义的服务。

自定义Ranger服务

定义服务类型

1. 创建JSON文件，用于描述服务组件，其中包含：

   • 资源名称，如：database, table, column等
   • 权限类型，如：select, update, create, drop等
   • 连接服务的配置信息，如：JDBC URL, JDBC driver, credentials等

   服务类型定义，服务定义文件的范例可在Ranger源码中查阅，一个完整的服务定义文件结构为：

   {
     "id":10,
     "name":"example",
     "implClass":"org.apache.ranger.services.RangerExampleService",
     "label":"Example",
     "description":"Example",
     "resources":[ ],
     "accessTypes":[ ],
     "configs":[ ],
     "enums": [ ],
     "contextEnrichers":[ ],
     "policyConditions":[ ]
   }

   文件中各个属性的值类型：

   • implClass是自定义授权插件类的全路径类名值（fully-qualified class name)，插件类需要继承RangerBaseService类，用于验证配置并进行资源查询，分别为RangerBaseService类的接口：

     public abstract HashMap<String, Object> validateConfig() throws Exception;
     public abstract List<String> lookupResource(ResourceLookupContext context) throws Exception;

   • resources定义服务中资源的类型

   • accessType定义在资源上进行授权的操作

   • configs设置用户需要配置的参数

   • id和name需要全局唯一，可以通过Ranger的查询接口查询现有服务
     curl -u xxx:xxx -X GET -H "Accept: application/json" -H "Content-Type: application/json" 'http://localhost:6080/service/public/v2/api/servicedef'

自定义服务发现

在Server端，实现服务发现，需要提供资源查找类，用户需要实现RangerBaseService，并将其注册到Ranger服务中。

public class RangerServiceYarn extends RangerBaseService {
	public HashMap<String, Object> validateConfig() throws Exception {
		...
    }
	public List<String> lookupResource(ResourceLookupContext context) throws Exception {
		...
    }
}

其中validateConfig函数配合服务定义时的配置说明，来验证用户服务配置的合法性。

Ranger提供自动补齐功能，用于查找资源，在Ranger Admin UI中查找资源时，会调用lookupResource函数。

服务注册

Ranger会加载默认服务配置，或通过API创建服务，因此，有以下两种方式来注册自定义服务：

编译到Ranger安装包

包含以下步骤

• 将服务定义文件添加到文件夹agents-common/src/main/resources/service-defs
• 修改EmbeddedServiceDefsUtil.java，添加自定义服务
• 将Ranger授权插件作为一个Ranger项目的一个模块，并将其添加到编译路径中
• 修改src/main/assembly/admin-web.xml，将自定义模块添加到Ranger Admin Webapp的集成中
• 编译Ranger，生成安装包，安装Ranger

通过Ranger接口注册

步骤如下

• 在Ranger安装目录下，创建目录*$RANGER_HOME/ews/webapp/WEB-INF/classes/ranger-plugins/myservice*，放入编译的自定义服务jar包
• 将服务定义文件提交到Ranger服务定义接口
  curl -u xxx:xxx -X POST -H "Accept: application/json" -H "Content-Type: application/json" –d @ranger-servicedef-yarn.json 'http://localhost:6080/service/public/v2/api/servicedef'

创建服务实例

服务注册后，可通过Ranger Admin或Ranger API创建服务实例，通过Ranger Admin，创建实例过程如下（HDFS）：

Label	Description
Service Name	Name of the Service, you will need to specify the service name in the agents config
Description	Give any description for reference
Active Status	You can choose this option to enable or disable the service
Username	Specify the end system user name that can be used for connection
Password	Add the password for the username above
hadoop.security.authentication	Specify the authentication type (Simple, Kerberos)
hbase.master.kerberos.principal	Specify the Kerberos principal for theHBase Master (Applicable only for Kerberos enabled environment)
hbase.security.authentication	Setting must match the hbase-site.xml setting for this property (Simple, Kerberos).
hbase.zookeeper.property.clientPort	Setting must match the hbase-site.xml setting for this property (default is : 2181).
hbase.zookeeper.quorum	Setting must match the hbase-site.xml setting for this property.
zookeeper.znode.parent	Setting must match the hbase-site.xml setting for this property.
Common Name for Certificate	Specify common name for certificate
Add New Configurations	Specify any other new configurations

其中的标签来自服务定义时的配置，Service Name会在后面自定义插件中使用到。

编写Ranger授权插件

在Plugin端，实现鉴权Authorizer，需要实现服务的授权hook，来对资源请求进行授权，此过程需要调用Ranger的API来进行验证及审计。hook应该在Plugin安装到hook时被注册。在处理授权请求时，需要使用到RangerBasePlugin或子类的实例，Authorizer实际是RangerBasePlugin的代理，实现时可使用代理模式或桥接器模式。以Yarn为例：

public class RangerYarnAuthorizer extends YarnAuthorizationProvider {
    private static volatile RangerYarnPlugin yarnPlugin = null;
    @Override
    public void init(Configuration conf) {
        yarnPlugin = new RangerYarnPlugin();
        yarnPlugin.init();
    }
    @Override
	public boolean checkPermission(AccessType accessType, PrivilegedEntity entity, UserGroupInformation ugi) {
        RangerAccessResourceImpl resource = new RangerAccessResourceImpl();
        resource.setValue("example-resource", identifier);

        RangerAccessRequestImpl request = new RangerAccessRequestImpl();
        request.setResource(resource);
        request.setAction(...);
        request.setAccessType(...);
        request.setUser(...);
        // request.setAccessTime(...);

        RangerAccessResult result = basePlugin.isAccessAllowed(
          rangerRequest, resultProcessor);

        if (result.getIsAllowed()) {
        	return true;
        } else {
			return false;
        }
    }
    @Override
	public boolean isAdmin(UserGroupInformation ugi) {
        ...
    }
    @Override
	public void setAdmins(AccessControlList acl, UserGroupInformation ugi) {
        ...
    }
    @Override
	public void setPermission(PrivilegedEntity entity, Map<AccessType, AccessControlList> permission, UserGroupInformation ugi) {
        ...
    }
}

RangerBasePlugin在对鉴权请求进行审核时，涉及到的信息包括审核用户、资源名、请求权限（Action并未使用）、请求时间（默认为当前时间），实际还包含请求服务名。用户可以实现RangerBasePlugin，以传递服务名或自定义服务行为，但建议使用RangerBasePlugin定义方法进行权限审核。

附：RangerBasePlugin源码简要分析

RangerBasePlugin是个功能复杂的客户端集合，其中包含了鉴权引擎、policy刷新器、Ranger管理客户端、Plugin注册中心等成员，核心函数包含

public class RangerBasePlugin {
    private static Map<String, RangerBasePlugin> servicePluginMap = new ConcurrentHashMap<>();								// init注册
    private PolicyRefresher           refresher;	// init初始化
	private RangerPolicyEngine        policyEngine;	// refresher刷新时初始化
    private AuditProviderFactory      auditProviderFactory;	// init初始化
    
    public RangerBasePlugin(String serviceType, String appId) {
		this.serviceType = serviceType;
		this.appId       = appId;
	}
    public RangerServiceDef getServiceDef() {
		RangerPolicyEngine policyEngine = this.policyEngine;
		return policyEngine != null ? policyEngine.getServiceDef() : null;
	}
    public AuditProviderFactory getAuditProviderFactory() { return auditProviderFactory; }
    // Warning: 重要，请保证在子类init函数中调用
    public void init() {
		RangerConfiguration configuration = RangerConfiguration.getInstance();
		configuration.addResourcesForServiceType(serviceType);

		String propertyPrefix    = "ranger.plugin." + serviceType;
        	// 默认30秒刷新policy
		long   pollingIntervalMs = configuration.getLong(propertyPrefix + ".policy.pollIntervalMs", 30 * 1000);
		String cacheDir          = configuration.get(propertyPrefix + ".policy.cache.dir");
		serviceName = configuration.get(propertyPrefix + ".service.name");
		clusterName = RangerConfiguration.getInstance().get(propertyPrefix + ".ambari.cluster.name", "");
		useForwardedIPAddress = configuration.getBoolean(propertyPrefix + ".use.x-forwarded-for.ipaddress", false);
		String trustedProxyAddressString = configuration.get(propertyPrefix + ".trusted.proxy.ipaddresses");
		trustedProxyAddresses = StringUtils.split(trustedProxyAddressString, RANGER_TRUSTED_PROXY_IPADDRESSES_SEPARATOR_CHAR);
		if (trustedProxyAddresses != null) 
			for (int i = 0; i < trustedProxyAddresses.length; i++) 
				trustedProxyAddresses[i] = trustedProxyAddresses[i].trim();

        	// 审计函数工厂
		if (configuration.getProperties() != null) {
			auditProviderFactory = new AuditProviderFactory();
			auditProviderFactory.init(configuration.getProperties(), appId);
		} 

		policyEngineOptions.configureForPlugin(configuration, propertyPrefix);
        	// 服务注册
		servicePluginMap.put(serviceName, this);
        	// Ranger管理客户端
		RangerAdminClient admin = createAdminClient(serviceName, appId, propertyPrefix);
		// Policy刷新器，Thread子类，调用startRefresher启动
		refresher = new PolicyRefresher(this, serviceType, appId, serviceName, admin, pollingIntervalMs, cacheDir);
		refresher.setDaemon(true);
		refresher.startRefresher();

		if (policyReorderIntervalMs >= 0 && policyReorderIntervalMs < 15 * 1000) 
			policyReorderIntervalMs = 15 * 1000;

		if (policyEngineOptions.disableTrieLookupPrefilter && policyReorderIntervalMs >0){
			policyEngineRefreshTimer = new Timer("PolicyEngineRefreshTimer", true);
			policyEngineRefreshTimer.schedule(new PolicyEngineRefresher(this),  policyReorderIntervalMs, policyReorderIntervalMs);
		}
	}
    
    // 此函数在refresher刷新时调用，用于初始化鉴权引擎
    public void setPolicies(ServicePolicies policies) {
        RangerPolicyEngine oldPolicyEngine = this.policyEngine;

        if (policies == null) {
            policies = getDefaultSvcPolicies();
        }
        if (policies == null) {
            this.policyEngine = null;
            readOnlyAuthContext = null;
        } else {
            currentAuthContext = new RangerAuthContext();
            RangerPolicyEngine policyEngine = new RangerPolicyEngineImpl(appId, policies, policyEngineOptions);
            policyEngine.setUseForwardedIPAddress(useForwardedIPAddress);
            policyEngine.setTrustedProxyAddresses(trustedProxyAddresses);
            this.policyEngine = policyEngine;
            currentAuthContext.setPolicyEngine(this.policyEngine);
            readOnlyAuthContext = new RangerAuthContext(currentAuthContext);
        }
        contextChanged();
    }
    
    public RangerAccessResult isAccessAllowed(RangerAccessRequest request, RangerAccessResultProcessor resultProcessor) {
	policyEngine.preProcess(request);
	return policyEngine.evaluatePolicies(request, RangerPolicy.POLICY_TYPE_ACCESS, resultProcessor);
    }
    
    public RangerAccessResult evalDataMaskPolicies(RangerAccessRequest request, RangerAccessResultProcessor resultProcessor) {
	RangerPolicyEngine policyEngine = this.policyEngine;
	policyEngine.preProcess(request);
	return policyEngine.evaluatePolicies(request, RangerPolicy.POLICY_TYPE_DATAMASK, resultProcessor);
    }

    public RangerAccessResult evalRowFilterPolicies(RangerAccessRequest request, RangerAccessResultProcessor resultProcessor) {
	policyEngine.preProcess(request);
	return policyEngine.evaluatePolicies(request, RangerPolicy.POLICY_TYPE_ROWFILTER, resultProcessor);
    }
    
    public RangerResourceAccessInfo getResourceAccessInfo(RangerAccessRequest request) {
	policyEngine.preProcess(request);
	return policyEngine.getResourceAccessInfo(request);
    }
    
    public void grantAccess(GrantRevokeRequest request, RangerAccessResultProcessor resultProcessor) throws Exception {
	refresher.getRangerAdminClient().grantAccess(request);
	auditGrantRevoke(request, "grant", isSuccess, resultProcessor);
    }

    public void revokeAccess(GrantRevokeRequest request, RangerAccessResultProcessor resultProcessor) throws Exception {
	admin.revokeAccess(request);
    }
    
    private ServicePolicies getDefaultSvcPolicies() {
	ServicePolicies ret = null;
	RangerServiceDef serviceDef = getServiceDef();
	if (serviceDef == null) {
	    serviceDef = getDefaultServiceDef();
	}
	if (serviceDef != null) {
	    ret = new ServicePolicies();
	    ret.setServiceDef(serviceDef);
	    ret.setServiceName(serviceName);
	    ret.setPolicies(new ArrayList<RangerPolicy>());
	}
	return ret;
    }
}

附：权限审计

审计是对权限行为的记录过程，贯穿RangerBasePlugin健全过程，默认通过solr进行审计，可能需要自定义服务端进行相关配置的调整，也可在调用isAccessAllowed时传入自定义审计函数。默认情况下，需要如下配置：

<property>
  <name>xasecure.audit.destination.solr</name>
  <value>true</value>
</property>

<property>
  <name>xasecure.audit.destination.solr.batch.filespool.dir</name>
  <value>/tmp/audit/solr/spool</value>
</property>

<property>
  <name>xasecure.audit.destination.solr.urls</name>
  <value>http://localhost:6083/solr/ranger_audits</value>
</property>

在Ranger插件使用

将Ranger插件安装到用户自定义服务中，需要将包含Authorizer的jar包包含到自定义服务的classpath，让其能够加载。同时，以下依赖的Ranger组件也必须包含：

• ranger-plugins-audit-<version>.jar
• ranger-plugins-common-<version>.jar
• ranger-plugins-cred-<version>.jar

Ranger插件在运行时需要使用到若干配置，classpath中也需要包含下列几个配置文件，这些文件会在Ranger插件启动时被搜索、加载

• ranger-<serviceType>-audit.xml
• ranger-<serviceType>-security.xml
• ranger-policymgr-ssl.xml

其中ranger-<serviceType>-security.xml定义了Plugin的一系列行为：

Configuration	Default Value	Comments
ranger.plugin.<serviceType>.service.name	无默认值，必须.	插件所拉取的权限policy的服务
ranger.plugin.<serviceType>.policy.source.impl	org.apache.ranger.admin.client.RangerAdminRESTClient	用于获取权限policy的类
ranger.plugin.<serviceType>.policy.rest.url	无默认值	RangerAdmin地址
ranger.plugin.<serviceType>.policy.rest.ssl.config.file	无默认值，如果需要在plugin与Ranger间提供SSL需要包含	SSL配置
ranger.plugin.<serviceType>.policy.cache.dir	权限默认缓存位置，如无，则不会进行权限缓存	缓存目录
ranger.plugin.<serviceType>.policy.pollIntervalMs	30000	权限刷新周期，ms

之后就可以在自定义服务中使用Authorizer进行权限审核了。